2024-03-25T16:16:27.121 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:16:27.121 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265 --verbose -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265/orig.config.yaml 2024-03-25T16:16:27.165 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:16:27.255 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621265' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi037.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNdzPNwpNxL8sLSxCBb3qYaQUyuKLhg/+dU6nOMhGbC99bahPbGjo7T9lDbLNGoTdxPzhvw8f2JmpIjH68p7bY= smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAx0A2u20CfmOD6mD1RM83qmHc3FrWuU9cHfiZY/crmiPZ9xhA8kxQyqM2MmBawk1+PAyifrIOL2XSXvt52zQjE= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:16:27.256 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:16:27.256 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:16:27.257 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:16:27.258 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:16:27.287 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:16:27.287 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:16:27.287 INFO:teuthology.packaging:ref: None 2024-03-25T16:16:27.287 INFO:teuthology.packaging:tag: None 2024-03-25T16:16:27.287 INFO:teuthology.packaging:branch: main 2024-03-25T16:16:27.287 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:16:27.288 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:16:27.754 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:16:27.755 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:16:27.812 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:16:27.812 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:16:27.833 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:16:27.852 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:16:27.865 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:16:27.887 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:11:20.937895', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNdzPNwpNxL8sLSxCBb3qYaQUyuKLhg/+dU6nOMhGbC99bahPbGjo7T9lDbLNGoTdxPzhvw8f2JmpIjH68p7bY='} 2024-03-25T16:16:27.910 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:11:20.939544', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAx0A2u20CfmOD6mD1RM83qmHc3FrWuU9cHfiZY/crmiPZ9xhA8kxQyqM2MmBawk1+PAyifrIOL2XSXvt52zQjE='} 2024-03-25T16:16:27.911 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:16:27.917 INFO:teuthology.task.internal:roles: ubuntu@smithi037.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:16:27.917 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:16:27.917 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:16:28.007 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f7f11dfa0d0>, signals=[15]) 2024-03-25T16:16:28.008 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:16:28.014 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:16:28.014 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi037.front.sepia.ceph.com 2024-03-25T16:16:28.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:16:28.126 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-03-25T16:16:28.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:16:28.216 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:16:28.226 DEBUG:teuthology.orchestra.run.smithi037:> uname -m 2024-03-25T16:16:28.244 INFO:teuthology.orchestra.run.smithi037.stdout:x86_64 2024-03-25T16:16:28.244 DEBUG:teuthology.orchestra.run.smithi037:> cat /etc/os-release 2024-03-25T16:16:28.302 INFO:teuthology.orchestra.run.smithi037.stdout:NAME="CentOS Stream" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION="9" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:ID="centos" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:VERSION_ID="9" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:ANSI_COLOR="0;31" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:16:28.303 INFO:teuthology.orchestra.run.smithi037.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:16:28.304 INFO:teuthology.lock.ops:Updating smithi037.front.sepia.ceph.com on lock server 2024-03-25T16:16:28.327 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-03-25T16:16:28.344 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-03-25T16:16:28.345 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-03-25T16:16:28.402 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-03-25T16:16:28.402 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-03-25T16:16:28.402 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-03-25T16:16:28.402 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:16:28.402 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-03-25T16:16:28.402 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:16:28.402 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:16:28.403 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-03-25T16:16:28.403 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:16:28.403 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:16:28.403 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:16:28.403 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:16:28.403 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:16:28.403 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:16:28.403 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-03-25T16:16:28.443 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:16:28.459 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:16:28.483 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:16:28.483 DEBUG:teuthology.orchestra.run.smithi037:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:16:28.486 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:16:28.502 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:16:28.510 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:16:28.510 DEBUG:teuthology.orchestra.run.smithi037:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:16:28.543 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:16:28.565 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:16:28.619 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:16:28.634 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:16:28.634 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:16:28.634 DEBUG:teuthology.orchestra.run.smithi037:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:16:28.635 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:16:28.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:16:28.652 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-25T16:16:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:16:28.654 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-03-25T16:16:28.714 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:16:28.715 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.14.0-361.el9.x86_64 2024-03-25T16:16:28.715 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum install -y kernel 2024-03-25T16:16:28.716 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:16:28.716 INFO:teuthology.task.kernel:Running kernel on smithi037: 5.14.0-361.el9.x86_64 2024-03-25T16:16:28.716 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum install -y kernel 2024-03-25T16:16:30.729 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-25T16:16:30.909 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-25T16:16:32.312 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 4.3 MB/s | 2.0 MB 00:00 2024-03-25T16:16:33.714 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 7.6 MB/s | 8.0 MB 00:01 2024-03-25T16:16:35.595 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-25T16:16:38.251 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - BaseOS 331 kB/s | 2.0 MB 00:06 2024-03-25T16:16:39.662 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 9.6 MB/s | 6.1 MB 00:00 2024-03-25T16:16:41.521 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 21 kB/s | 16 kB 00:00 2024-03-25T16:16:43.107 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.0 MB 00:04 2024-03-25T16:16:43.641 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 10 MB/s | 21 MB 00:02 2024-03-25T16:16:44.808 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 31 MB/s | 19 MB 00:00 2024-03-25T16:16:48.638 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-25T16:16:48.950 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.1 MB 00:00 2024-03-25T16:16:50.385 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-03-25T16:16:50.778 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:16:50.778 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:16:50.916 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:16:50.927 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:16:50.927 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-03-25T16:16:50.927 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:16:50.927 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:16:50.927 INFO:teuthology.orchestra.run.smithi117.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:16:50.927 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-25T16:16:50.927 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:16:50.927 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:16:50.927 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:16:50.927 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:16:50.927 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:16:50.928 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:16:50.928 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-03-25T16:16:50.928 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:16:50.928 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 76 M 2024-03-25T16:16:50.928 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 123 M 2024-03-25T16:16:50.928 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:16:51.164 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 401 kB/s | 44 kB 00:00 2024-03-25T16:16:51.286 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-03-25T16:16:51.765 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-25T16:16:52.756 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): kernel-modules-core-5.14.0-430.el9.x86_6 17 MB/s | 27 MB 00:01 2024-03-25T16:16:53.090 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 16 MB/s | 33 MB 00:02 2024-03-25T16:16:53.090 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:16:53.090 INFO:teuthology.orchestra.run.smithi117.stdout:Total 35 MB/s | 76 MB 00:02 2024-03-25T16:16:53.758 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:16:54.125 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:16:54.125 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:16:55.128 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:16:55.128 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:16:56.428 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-25T16:16:57.690 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:16:58.299 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:16:58.381 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:16:58.803 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:16:58.803 INFO:teuthology.orchestra.run.smithi037.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:16:58.895 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:16:58.902 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:16:58.903 INFO:teuthology.orchestra.run.smithi037.stdout:Install 4 Packages 2024-03-25T16:16:58.903 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:16:58.903 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 76 M 2024-03-25T16:16:58.903 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 123 M 2024-03-25T16:16:58.903 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:16:59.196 INFO:teuthology.orchestra.run.smithi037.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 177 kB/s | 44 kB 00:00 2024-03-25T16:17:01.113 INFO:teuthology.orchestra.run.smithi037.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 6.9 MB/s | 15 MB 00:02 2024-03-25T16:17:01.117 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:17:01.259 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:17:01.663 INFO:teuthology.orchestra.run.smithi037.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 12 MB/s | 33 MB 00:02 2024-03-25T16:17:02.247 INFO:teuthology.orchestra.run.smithi037.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 9.0 MB/s | 27 MB 00:03 2024-03-25T16:17:02.247 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:17:02.247 INFO:teuthology.orchestra.run.smithi037.stdout:Total 23 MB/s | 76 MB 00:03 2024-03-25T16:17:02.942 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:17:03.375 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:17:03.375 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:17:04.434 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:17:04.434 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:17:05.930 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:17:05.967 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:07.084 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:17:07.719 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:17:07.814 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:17:10.710 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:10.712 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:17:10.843 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:17:15.940 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:17:15.985 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:21.113 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:40.186 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:40.186 INFO:teuthology.orchestra.run.smithi117.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:17:40.186 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:17:40.230 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:40.635 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:40.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:17:40.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:17:40.636 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:17:41.014 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:41.014 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:17:41.015 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:17:41.015 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:41.015 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:17:41.015 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:17:41.015 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:17:41.015 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:17:41.015 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:17:41.344 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:17:41.438 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:41.438 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:17:41.438 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:17:41.439 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:17:41.439 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:17:41.439 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:17:41.439 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:17:41.439 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:17:41.519 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:41.519 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:17:41.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -qi grub2-tools 2024-03-25T16:17:41.553 INFO:teuthology.orchestra.run.smithi117.stdout:Name : grub2-tools 2024-03-25T16:17:41.553 INFO:teuthology.orchestra.run.smithi117.stdout:Epoch : 1 2024-03-25T16:17:41.553 INFO:teuthology.orchestra.run.smithi117.stdout:Version : 2.06 2024-03-25T16:17:41.553 INFO:teuthology.orchestra.run.smithi117.stdout:Release : 68.el9 2024-03-25T16:17:41.553 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: x86_64 2024-03-25T16:17:41.553 INFO:teuthology.orchestra.run.smithi117.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:17:41.553 INFO:teuthology.orchestra.run.smithi117.stdout:Group : Unspecified 2024-03-25T16:17:41.553 INFO:teuthology.orchestra.run.smithi117.stdout:Size : 8264254 2024-03-25T16:17:41.553 INFO:teuthology.orchestra.run.smithi117.stdout:License : GPLv3+ 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:Packager : builder@centos.org 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:Vendor : CentOS 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:Summary : Support tools for GRUB. 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:Description : 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout:hardware devices. 2024-03-25T16:17:41.554 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:17:41.555 INFO:teuthology.orchestra.run.smithi117.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:17:41.557 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:17:41.557 INFO:teuthology.task.kernel:Updating grub on smithi117 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:17:41.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:17:42.136 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-03-25T16:17:43.535 INFO:teuthology.orchestra.run.smithi117.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:17:43.568 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-03-25T16:17:43.570 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-25T16:17:43.585 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.nRRgJ6lEQG 2024-03-25T16:17:43.586 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.nRRgJ6lEQG 2024-03-25T16:17:43.656 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.nRRgJ6lEQG 2024-03-25T16:17:43.792 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.nRRgJ6lEQG is 6KB 2024-03-25T16:17:43.840 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.nRRgJ6lEQG 2024-03-25T16:17:43.854 DEBUG:teuthology.orchestra.run.smithi117:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:17:43.921 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:17:43.921 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:17:43.922 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:17:43.922 INFO:teuthology.orchestra.run.smithi117.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:17:43.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:17:44.078 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-03-25T16:17:52.096 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:52.097 INFO:teuthology.orchestra.run.smithi037.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:17:52.097 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:17:52.142 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:52.558 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:52.558 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:17:52.558 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:17:52.559 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:17:53.097 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:17:53.097 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:17:53.098 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:17:53.098 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:53.098 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:17:53.098 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:17:53.098 INFO:teuthology.orchestra.run.smithi037.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:17:53.098 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:17:53.098 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:17:53.420 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:17:53.521 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:53.521 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:17:53.521 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:17:53.521 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:17:53.521 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:17:53.521 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:17:53.521 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi037.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:17:53.521 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:17:53.604 INFO:teuthology.orchestra.run.smithi037.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:17:53.605 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:17:53.605 DEBUG:teuthology.orchestra.run.smithi037:> sudo rpm -qi grub2-tools 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:Name : grub2-tools 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:Epoch : 1 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:Version : 2.06 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:Release : 68.el9 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:Architecture: x86_64 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:Group : Unspecified 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:Size : 8264254 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:License : GPLv3+ 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:17:53.643 INFO:teuthology.orchestra.run.smithi037.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout:Packager : builder@centos.org 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout:Vendor : CentOS 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout:Summary : Support tools for GRUB. 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout:Description : 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout:hardware devices. 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:17:53.644 INFO:teuthology.orchestra.run.smithi037.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:17:53.645 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:17:53.645 INFO:teuthology.task.kernel:Updating grub on smithi037 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:17:53.645 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:17:54.217 INFO:teuthology.orchestra.run.smithi037.stderr:Generating grub configuration file ... 2024-03-25T16:17:55.628 INFO:teuthology.orchestra.run.smithi037.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:17:55.661 INFO:teuthology.orchestra.run.smithi037.stderr:done 2024-03-25T16:17:55.663 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-03-25T16:17:55.679 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.J86jkDmnrr 2024-03-25T16:17:55.680 DEBUG:teuthology.orchestra.run.smithi037:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.J86jkDmnrr 2024-03-25T16:17:55.751 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.J86jkDmnrr 2024-03-25T16:17:55.896 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.J86jkDmnrr is 6KB 2024-03-25T16:17:55.945 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.J86jkDmnrr 2024-03-25T16:17:55.959 DEBUG:teuthology.orchestra.run.smithi037:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:17:56.036 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:17:56.036 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:17:56.036 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:17:56.036 INFO:teuthology.orchestra.run.smithi037.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:17:56.038 DEBUG:teuthology.orchestra.run.smithi037:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:17:56.192 DEBUG:teuthology.orchestra.run.smithi037:> sudo shutdown -r now 2024-03-25T16:18:14.098 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:18:14.099 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-25T16:18:14.100 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:18:26.206 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:18:26.206 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-25T16:18:26.207 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:00.767 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-03-25T16:19:09.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-25T16:19:09.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:12.830 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.37 2024-03-25T16:19:14.102 DEBUG:teuthology.orchestra.remote:timed out 2024-03-25T16:19:23.110 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-25T16:19:23.111 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:24.365 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-25T16:19:24.831 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-25T16:19:24.832 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:19:24.950 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-25T16:19:24.950 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:19:24.950 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-03-25T16:19:24.964 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:19:24.964 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:19:24.964 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:19:24.965 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:19:25.235 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-03-25T16:19:25.715 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-25T16:19:25.715 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:19:25.715 DEBUG:teuthology.orchestra.run.smithi037:> uname -r 2024-03-25T16:19:25.729 INFO:teuthology.orchestra.run.smithi037.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:19:25.729 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:19:25.729 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:19:25.729 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:19:25.966 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:19:25.966 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:19:26.061 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2024-03-25T16:19:26.101 DEBUG:teuthology.parallel:result is None 2024-03-25T16:19:26.730 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:19:26.731 DEBUG:teuthology.orchestra.run.smithi037:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:19:26.823 INFO:teuthology.orchestra.run.smithi037.stdout:ttyS1 2024-03-25T16:19:26.862 DEBUG:teuthology.parallel:result is None 2024-03-25T16:19:26.862 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:19:26.869 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:19:26.870 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:19:26.872 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:19:26.887 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:19:26.909 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:19:26.942 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:19:26.943 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:19:26.945 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:19:26.970 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:19:26.977 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:19:26.977 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:19:27.008 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:19:27.034 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:19:27.047 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:19:27.059 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:19:27.073 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:19:27.074 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:19:27.081 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:19:27.081 DEBUG:teuthology.orchestra.run.smithi037:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:19:27.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:19:27.118 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:19:27.127 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:19:27.127 DEBUG:teuthology.orchestra.run.smithi037:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:19:27.159 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:19:27.174 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:19:27.255 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:19:27.298 DEBUG:teuthology.orchestra.run.smithi037:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:19:27.381 DEBUG:teuthology.orchestra.run.smithi037:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:19:27.421 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T16:19:27.421 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:19:27.488 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:19:27.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:19:27.578 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:19:27.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:19:27.703 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T16:19:27.703 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:19:27.774 DEBUG:teuthology.orchestra.run.smithi037:> sudo service rsyslog restart 2024-03-25T16:19:27.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-03-25T16:19:27.814 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:19:27.867 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:19:28.344 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:19:28.351 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:19:28.352 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:19:28.405 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:19:28.449 DEBUG:teuthology.orchestra.run.smithi037:> sudo service auditd rotate 2024-03-25T16:19:28.508 INFO:teuthology.orchestra.run.smithi037.stdout:Rotating logs: 2024-03-25T16:19:28.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-03-25T16:19:28.555 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-03-25T16:19:28.557 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:19:28.557 DEBUG:teuthology.orchestra.run.smithi037:> /usr/sbin/getenforce 2024-03-25T16:19:28.573 INFO:teuthology.orchestra.run.smithi037.stdout:Permissive 2024-03-25T16:19:28.573 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-03-25T16:19:28.591 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-03-25T16:19:28.591 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi037.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi117.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:19:28.592 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:19:28.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:19:28.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:19:28.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:19:28.671 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:19:28.671 DEBUG:teuthology.orchestra.run.smithi037:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:19:28.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:19:28.752 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:19:28.762 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:19:28.780 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:19:28.781 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi037.front.sepia.ceph.com,smithi117.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:23:47.795 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi037.front.sepia.ceph.com'), Remote(name='ubuntu@smithi117.front.sepia.ceph.com')] 2024-03-25T16:23:47.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-25T16:23:47.797 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi037.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:47.890 DEBUG:teuthology.orchestra.run.smithi037:> true 2024-03-25T16:23:47.969 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi037.front.sepia.ceph.com' 2024-03-25T16:23:47.969 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-25T16:23:47.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:23:48.063 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-03-25T16:23:48.136 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-03-25T16:23:48.137 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:23:48.150 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:23:48.151 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:23:48.151 DEBUG:teuthology.orchestra.run.smithi037:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:23:48.154 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:23:48.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:23:48.183 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:23:48.200 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:23:48.206 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:23:48.223 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:23:48.230 INFO:teuthology.orchestra.run.smithi037.stderr:sudo: ntpd: command not found 2024-03-25T16:23:48.244 INFO:teuthology.orchestra.run.smithi037.stdout:506 Cannot talk to daemon 2024-03-25T16:23:48.252 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-03-25T16:23:48.261 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:23:48.266 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-03-25T16:23:48.277 INFO:teuthology.orchestra.run.smithi037.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:23:48.282 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:23:48.298 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:23:48.335 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:23:48.339 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:23:48.339 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-03-25T16:23:48.339 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:48.339 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:48.340 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:48.340 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:48.349 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:23:48.352 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:23:48.352 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-25T16:23:48.353 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:48.353 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:48.353 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:48.353 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:23:48.354 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:23:48.366 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:23:48.366 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:23:48.366 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:23:48.366 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:23:48.373 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:23:48.374 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:23:48.374 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:23:48.375 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:23:48.377 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:23:48.377 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:23:48.528 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:23:48.529 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:23:48.599 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:23:48.599 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:23:48.920 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:23:48.921 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T16:23:48.921 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:23:48.929 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:23:48.930 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T16:23:48.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:23:48.951 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:23:48.952 DEBUG:teuthology.orchestra.run.smithi037:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:23:48.959 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:23:48.960 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:23:49.021 DEBUG:teuthology.orchestra.run.smithi037:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:23:49.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:23:49.102 DEBUG:teuthology.orchestra.run.smithi037:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:23:49.110 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:23:49.168 INFO:teuthology.orchestra.run.smithi037.stdout:check_obsoletes = 1 2024-03-25T16:23:49.170 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum clean all 2024-03-25T16:23:49.175 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-03-25T16:23:49.177 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-03-25T16:23:49.416 INFO:teuthology.orchestra.run.smithi117.stdout:74 files removed 2024-03-25T16:23:49.422 INFO:teuthology.orchestra.run.smithi037.stdout:74 files removed 2024-03-25T16:23:49.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw 2024-03-25T16:23:49.449 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-radosgw 2024-03-25T16:23:50.645 INFO:teuthology.orchestra.run.smithi037.stdout:ceph packages for x86_64 92 kB/s | 82 kB 00:00 2024-03-25T16:23:50.694 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 86 kB/s | 82 kB 00:00 2024-03-25T16:23:51.487 INFO:teuthology.orchestra.run.smithi037.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:23:51.528 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:23:52.304 INFO:teuthology.orchestra.run.smithi037.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:23:52.403 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:23:52.723 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T16:23:52.847 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-25T16:23:54.207 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS-9 - BaseOS 6.2 MB/s | 2.0 MB 00:00 2024-03-25T16:23:54.297 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-25T16:23:55.040 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - BaseOS 16 MB/s | 8.0 MB 00:00 2024-03-25T16:23:55.607 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.0 MB 00:00 2024-03-25T16:23:58.191 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 19 MB 00:02 2024-03-25T16:24:00.006 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 5.6 MB/s | 19 MB 00:03 2024-03-25T16:24:03.175 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.1 MB 00:01 2024-03-25T16:24:04.622 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 5.2 MB/s | 6.1 MB 00:01 2024-03-25T16:24:04.976 INFO:teuthology.orchestra.run.smithi037.stdout:CentOS Stream 9 - Extras packages 25 kB/s | 16 kB 00:00 2024-03-25T16:24:06.202 INFO:teuthology.orchestra.run.smithi037.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-25T16:24:07.248 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 12 kB/s | 16 kB 00:01 2024-03-25T16:24:10.182 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 7.4 MB/s | 21 MB 00:02 2024-03-25T16:24:11.277 INFO:teuthology.orchestra.run.smithi037.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-03-25T16:24:13.808 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:24:13.938 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:24:13.939 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-25T16:24:13.939 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:24:13.939 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:24:13.939 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:24:13.939 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrading: 2024-03-25T16:24:13.939 INFO:teuthology.orchestra.run.smithi037.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:24:13.939 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:24:13.939 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-25T16:24:13.939 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options 2024-03-25T16:24:13.939 INFO:teuthology.orchestra.run.smithi037.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:24:13.940 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:24:13.941 INFO:teuthology.orchestra.run.smithi037.stdout:Installing weak dependencies: 2024-03-25T16:24:13.942 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:24:13.942 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:24:13.942 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:24:13.942 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:24:13.942 INFO:teuthology.orchestra.run.smithi037.stdout:Install 30 Packages 2024-03-25T16:24:13.942 INFO:teuthology.orchestra.run.smithi037.stdout:Upgrade 2 Packages 2024-03-25T16:24:13.942 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:24:13.943 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 58 M 2024-03-25T16:24:13.943 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:24:15.106 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-25T16:24:17.535 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:24:17.538 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:24:17.538 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-25T16:24:17.538 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:24:17.539 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:24:17.540 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:24:17.541 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:24:17.542 INFO:teuthology.orchestra.run.smithi117.stdout:Install 30 Packages 2024-03-25T16:24:17.542 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 2 Packages 2024-03-25T16:24:17.542 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:24:17.542 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 58 M 2024-03-25T16:24:17.542 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:24:25.806 INFO:teuthology.orchestra.run.smithi117.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 700 kB/s | 5.3 MB 00:07 2024-03-25T16:24:25.898 INFO:teuthology.orchestra.run.smithi117.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 25 kB 00:00 2024-03-25T16:24:26.940 INFO:teuthology.orchestra.run.smithi117.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 712 kB/s | 742 kB 00:01 2024-03-25T16:24:27.303 INFO:teuthology.orchestra.run.smithi037.stdout:(1/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 752 kB/s | 9.7 MB 00:13 2024-03-25T16:24:27.403 INFO:teuthology.orchestra.run.smithi037.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 252 kB/s | 25 kB 00:00 2024-03-25T16:24:27.815 INFO:teuthology.orchestra.run.smithi117.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 570 kB/s | 498 kB 00:00 2024-03-25T16:24:28.437 INFO:teuthology.orchestra.run.smithi037.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 718 kB/s | 742 kB 00:01 2024-03-25T16:24:29.253 INFO:teuthology.orchestra.run.smithi037.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 610 kB/s | 498 kB 00:00 2024-03-25T16:24:30.054 INFO:teuthology.orchestra.run.smithi037.stdout:(5/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 342 kB/s | 5.3 MB 00:16 2024-03-25T16:24:30.279 INFO:teuthology.orchestra.run.smithi037.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 200 kB/s | 45 kB 00:00 2024-03-25T16:24:30.771 INFO:teuthology.orchestra.run.smithi037.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 267 kB/s | 131 kB 00:00 2024-03-25T16:24:31.804 INFO:teuthology.orchestra.run.smithi037.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 157 kB/s | 162 kB 00:01 2024-03-25T16:24:33.229 INFO:teuthology.orchestra.run.smithi037.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 225 kB/s | 320 kB 00:01 2024-03-25T16:24:33.847 INFO:teuthology.orchestra.run.smithi117.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 866 kB/s | 5.1 MB 00:06 2024-03-25T16:24:34.196 INFO:teuthology.orchestra.run.smithi037.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 308 kB/s | 298 kB 00:00 2024-03-25T16:24:34.281 INFO:teuthology.orchestra.run.smithi117.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 104 kB/s | 45 kB 00:00 2024-03-25T16:24:34.706 INFO:teuthology.orchestra.run.smithi117.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 309 kB/s | 131 kB 00:00 2024-03-25T16:24:34.897 INFO:teuthology.orchestra.run.smithi037.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 142 kB/s | 99 kB 00:00 2024-03-25T16:24:35.005 INFO:teuthology.orchestra.run.smithi037.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 978 kB/s | 106 kB 00:00 2024-03-25T16:24:35.230 INFO:teuthology.orchestra.run.smithi037.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 707 kB/s | 159 kB 00:00 2024-03-25T16:24:35.264 INFO:teuthology.orchestra.run.smithi037.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:24:35.315 INFO:teuthology.orchestra.run.smithi117.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 266 kB/s | 162 kB 00:00 2024-03-25T16:24:35.323 INFO:teuthology.orchestra.run.smithi037.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:24:35.365 INFO:teuthology.orchestra.run.smithi037.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-25T16:24:35.406 INFO:teuthology.orchestra.run.smithi037.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 991 kB/s | 41 kB 00:00 2024-03-25T16:24:35.632 INFO:teuthology.orchestra.run.smithi037.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 962 kB/s | 216 kB 00:00 2024-03-25T16:24:35.691 INFO:teuthology.orchestra.run.smithi037.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 599 kB/s | 35 kB 00:00 2024-03-25T16:24:35.749 INFO:teuthology.orchestra.run.smithi037.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 763 kB/s | 44 kB 00:00 2024-03-25T16:24:36.024 INFO:teuthology.orchestra.run.smithi037.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.1 MB/s | 309 kB 00:00 2024-03-25T16:24:36.173 INFO:teuthology.orchestra.run.smithi117.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 373 kB/s | 320 kB 00:00 2024-03-25T16:24:36.958 INFO:teuthology.orchestra.run.smithi037.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.7 MB/s | 4.4 MB 00:00 2024-03-25T16:24:37.040 INFO:teuthology.orchestra.run.smithi117.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 344 kB/s | 298 kB 00:00 2024-03-25T16:24:37.042 INFO:teuthology.orchestra.run.smithi037.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 312 kB/s | 26 kB 00:00 2024-03-25T16:24:37.125 INFO:teuthology.orchestra.run.smithi037.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 585 kB/s | 49 kB 00:00 2024-03-25T16:24:37.217 INFO:teuthology.orchestra.run.smithi037.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 735 kB/s | 67 kB 00:00 2024-03-25T16:24:37.373 INFO:teuthology.orchestra.run.smithi117.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 298 kB/s | 99 kB 00:00 2024-03-25T16:24:37.434 INFO:teuthology.orchestra.run.smithi037.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.8 MB/s | 838 kB 00:00 2024-03-25T16:24:37.510 INFO:teuthology.orchestra.run.smithi037.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.5 MB/s | 192 kB 00:00 2024-03-25T16:24:37.532 INFO:teuthology.orchestra.run.smithi117.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 668 kB/s | 106 kB 00:00 2024-03-25T16:24:37.632 INFO:teuthology.orchestra.run.smithi117.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 159 kB 00:00 2024-03-25T16:24:37.674 INFO:teuthology.orchestra.run.smithi117.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:24:37.754 INFO:teuthology.orchestra.run.smithi037.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-03-25T16:24:37.783 INFO:teuthology.orchestra.run.smithi117.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 6.0 MB/s | 661 kB 00:00 2024-03-25T16:24:37.825 INFO:teuthology.orchestra.run.smithi117.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:24:37.850 INFO:teuthology.orchestra.run.smithi117.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.6 MB/s | 41 kB 00:00 2024-03-25T16:24:38.075 INFO:teuthology.orchestra.run.smithi117.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 963 kB/s | 216 kB 00:00 2024-03-25T16:24:38.125 INFO:teuthology.orchestra.run.smithi117.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 700 kB/s | 35 kB 00:00 2024-03-25T16:24:38.176 INFO:teuthology.orchestra.run.smithi117.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 893 kB/s | 44 kB 00:00 2024-03-25T16:24:38.409 INFO:teuthology.orchestra.run.smithi117.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-03-25T16:24:38.676 INFO:teuthology.orchestra.run.smithi117.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-03-25T16:24:38.727 INFO:teuthology.orchestra.run.smithi117.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 524 kB/s | 26 kB 00:00 2024-03-25T16:24:38.768 INFO:teuthology.orchestra.run.smithi117.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 1.1 MB/s | 49 kB 00:00 2024-03-25T16:24:38.827 INFO:teuthology.orchestra.run.smithi117.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-25T16:24:38.894 INFO:teuthology.orchestra.run.smithi117.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-25T16:24:39.036 INFO:teuthology.orchestra.run.smithi117.stdout:(27/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 474 kB/s | 9.7 MB 00:21 2024-03-25T16:24:39.061 INFO:teuthology.orchestra.run.smithi117.stdout:(28/32): re2-20211101-3.el9.x86_64.rpm 1.1 MB/s | 192 kB 00:00 2024-03-25T16:24:39.370 INFO:teuthology.orchestra.run.smithi117.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 4.7 MB/s | 1.6 MB 00:00 2024-03-25T16:24:43.144 INFO:teuthology.orchestra.run.smithi037.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 376 kB/s | 5.1 MB 00:13 2024-03-25T16:24:46.352 INFO:teuthology.orchestra.run.smithi117.stdout:(30/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 474 kB/s | 3.4 MB 00:07 2024-03-25T16:24:47.252 INFO:teuthology.orchestra.run.smithi037.stdout:(30/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 613 kB/s | 20 MB 00:33 2024-03-25T16:24:47.744 INFO:teuthology.orchestra.run.smithi117.stdout:(31/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 383 kB/s | 3.1 MB 00:08 2024-03-25T16:24:49.685 INFO:teuthology.orchestra.run.smithi037.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 289 kB/s | 3.4 MB 00:11 2024-03-25T16:24:50.502 INFO:teuthology.orchestra.run.smithi037.stdout:(32/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 435 kB/s | 3.1 MB 00:07 2024-03-25T16:24:50.505 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:50.505 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.6 MB/s | 58 MB 00:36 2024-03-25T16:24:50.535 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:24:50.574 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:24:50.574 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:24:51.067 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:24:51.067 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:24:52.704 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:24:52.805 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:24:52.848 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:24:52.874 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:24:52.905 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:24:52.921 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:24:53.138 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:24:53.204 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:24:53.610 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:24:53.650 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:24:53.743 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:24:53.832 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:24:53.860 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:24:53.922 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:24:53.951 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:24:53.996 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:24:54.065 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:24:54.096 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:24:54.141 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:24:54.177 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:24:54.289 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:24:54.321 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:24:54.361 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:24:54.394 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:24:54.423 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:24:54.455 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:24:54.785 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:24:54.810 INFO:teuthology.orchestra.run.smithi037.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:24:54.900 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:24:54.931 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:24:54.975 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:24:55.492 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:24:55.529 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:24:55.600 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:24:55.672 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:24:55.706 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:24:57.963 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:24:58.003 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:24:58.580 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:24:58.623 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:24:58.711 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:24:58.711 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:24:58.711 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:24:58.711 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:24:58.757 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:07.091 INFO:teuthology.orchestra.run.smithi117.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 415 kB/s | 20 MB 00:49 2024-03-25T16:25:07.094 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:07.094 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.2 MB/s | 58 MB 00:49 2024-03-25T16:25:07.124 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:25:07.161 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:25:07.161 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:25:07.632 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:25:07.632 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:25:08.296 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:08.296 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /sys 2024-03-25T16:25:08.297 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /proc 2024-03-25T16:25:08.297 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /mnt 2024-03-25T16:25:08.297 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /var/tmp 2024-03-25T16:25:08.297 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /home 2024-03-25T16:25:08.297 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /root 2024-03-25T16:25:08.297 INFO:teuthology.orchestra.run.smithi037.stdout:skipping the directory /tmp 2024-03-25T16:25:08.297 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:25:09.110 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:25:09.218 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:25:09.236 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:25:09.253 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:25:09.260 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:25:09.260 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:09.260 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:25:09.261 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:25:09.261 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:25:09.261 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:25:09.263 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:25:09.279 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:25:09.302 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:25:09.318 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:25:09.348 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:25:09.348 INFO:teuthology.orchestra.run.smithi037.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:09.519 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:25:09.573 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:25:09.891 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:25:09.950 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:10.055 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:10.142 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:25:10.165 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:25:10.210 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:25:10.239 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:25:10.283 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:25:10.351 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:25:10.376 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:25:10.438 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:25:10.465 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:25:10.573 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:25:10.608 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:25:10.623 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:10.623 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:25:10.623 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:25:10.623 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:25:10.623 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:25:10.623 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:25:10.623 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:25:10.623 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:10.623 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:25:10.623 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:25:10.624 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:25:10.625 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:25:10.625 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:25:10.625 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:25:10.625 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:25:10.625 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:25:10.625 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:25:10.625 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:25:10.625 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:10.625 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:25:10.625 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:25:10.649 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:25:10.681 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:25:10.710 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:25:10.743 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:25:11.061 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:25:11.089 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:25:11.170 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:25:11.202 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:25:11.242 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:25:11.692 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout:Upgraded: 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:25:11.693 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:25:11.694 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:25:11.695 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:25:11.696 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:25:11.725 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:25:11.805 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:25:11.872 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:25:11.902 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:25:11.979 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-test 2024-03-25T16:25:12.547 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:01:01 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:25:12.702 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:25:12.703 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:25:12.703 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-25T16:25:12.703 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:25:12.703 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:25:12.703 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:25:12.703 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-25T16:25:12.704 INFO:teuthology.orchestra.run.smithi037.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:25:12.704 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:25:12.704 INFO:teuthology.orchestra.run.smithi037.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:25:12.704 INFO:teuthology.orchestra.run.smithi037.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:25:12.704 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:25:12.704 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:25:12.704 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:25:12.705 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:25:12.705 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-03-25T16:25:12.705 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:25:12.706 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 48 M 2024-03-25T16:25:12.706 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 204 M 2024-03-25T16:25:12.706 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:25:12.953 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.0 MB/s | 217 kB 00:00 2024-03-25T16:25:13.036 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.1 MB/s | 300 kB 00:00 2024-03-25T16:25:13.078 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 63 kB 00:00 2024-03-25T16:25:13.220 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.3 MB/s | 187 kB 00:00 2024-03-25T16:25:13.478 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 224 kB/s | 163 kB 00:00 2024-03-25T16:25:14.116 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:14.157 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:14.700 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:25:14.735 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:25:14.809 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:25:14.809 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:25:14.809 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:25:14.809 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:25:14.853 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:24.150 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:24.150 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-03-25T16:25:24.150 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-03-25T16:25:24.150 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-03-25T16:25:24.150 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-03-25T16:25:24.150 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-03-25T16:25:24.151 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-03-25T16:25:24.151 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-03-25T16:25:24.151 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:25:25.022 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:25:25.045 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:25:25.045 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:25.046 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:25:25.046 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:25:25.046 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:25:25.046 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:25:25.047 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:25:25.115 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:25:25.115 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:26.282 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:26.282 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:25:26.282 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:25:26.282 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:25:26.282 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:25:26.282 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:25:26.282 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:25:26.282 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:25:26.283 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:25:26.284 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:25:26.284 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:25:26.284 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:25:26.284 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:25:26.284 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:25:26.284 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:25:26.284 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:25:26.285 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:25:26.285 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:25:26.285 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:25:26.285 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:25:26.285 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:25:26.285 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:25:27.231 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:25:27.232 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:25:27.233 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:25:27.234 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:25:27.235 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:25:27.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-test 2024-03-25T16:25:28.070 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:13 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:25:28.222 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:25:28.223 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:25:28.224 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:25:28.224 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:25:28.224 INFO:teuthology.orchestra.run.smithi117.stdout:Install 6 Packages 2024-03-25T16:25:28.224 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:25:28.224 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 48 M 2024-03-25T16:25:28.224 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 204 M 2024-03-25T16:25:28.224 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:25:28.436 INFO:teuthology.orchestra.run.smithi117.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:25:28.478 INFO:teuthology.orchestra.run.smithi117.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-25T16:25:28.511 INFO:teuthology.orchestra.run.smithi117.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:25:28.578 INFO:teuthology.orchestra.run.smithi117.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-25T16:25:29.045 INFO:teuthology.orchestra.run.smithi117.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 210 kB/s | 163 kB 00:00 2024-03-25T16:27:13.950 INFO:teuthology.orchestra.run.smithi117.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 458 kB/s | 47 MB 01:45 2024-03-25T16:27:13.952 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:13.952 INFO:teuthology.orchestra.run.smithi117.stdout:Total 467 kB/s | 48 MB 01:45 2024-03-25T16:27:13.964 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:27:13.984 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:27:13.984 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:27:14.370 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:27:14.370 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:27:14.957 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:27:15.025 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:27:15.065 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:27:15.105 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:27:15.157 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:27:15.186 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:27:16.492 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 391 kB/s | 47 MB 02:03 2024-03-25T16:27:16.494 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:16.494 INFO:teuthology.orchestra.run.smithi037.stdout:Total 399 kB/s | 48 MB 02:03 2024-03-25T16:27:16.505 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:27:16.526 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:27:16.526 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:27:16.916 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:27:16.917 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:27:17.520 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:27:17.600 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:27:17.637 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:27:17.677 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:27:17.728 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:27:17.752 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:27:19.638 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:27:19.719 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:27:22.099 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:27:22.099 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:27:22.099 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:27:22.099 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:27:22.099 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:27:22.099 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:27:22.511 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:27:22.512 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:27:22.512 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:27:22.512 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:22.512 INFO:teuthology.orchestra.run.smithi117.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:27:22.512 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:22.512 INFO:teuthology.orchestra.run.smithi117.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:27:22.512 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:27:22.512 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:27:22.512 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:27:22.512 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:27:22.572 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:27:22.646 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:27:22.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph 2024-03-25T16:27:23.352 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:03:08 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:27:23.506 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:27:23.508 INFO:teuthology.orchestra.run.smithi117.stdout:========================================================================================= 2024-03-25T16:27:23.508 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-25T16:27:23.508 INFO:teuthology.orchestra.run.smithi117.stdout:========================================================================================= 2024-03-25T16:27:23.508 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:27:23.508 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:27:23.508 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:27:23.509 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:27:23.510 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout:========================================================================================= 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout:Install 36 Packages 2024-03-25T16:27:23.511 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:27:23.512 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 30 M 2024-03-25T16:27:23.512 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 107 M 2024-03-25T16:27:23.512 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:27:24.188 INFO:teuthology.orchestra.run.smithi117.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-25T16:27:25.196 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:27:25.218 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:27:25.218 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:27:25.218 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:27:25.218 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:27:25.218 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:27:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:27:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:27:25.587 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:27:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:27:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:27:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:27:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:27:25.587 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:27:25.588 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:27:25.842 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph 2024-03-25T16:27:26.413 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:03:15 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:27:26.571 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:27:26.574 INFO:teuthology.orchestra.run.smithi037.stdout:========================================================================================= 2024-03-25T16:27:26.574 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-25T16:27:26.574 INFO:teuthology.orchestra.run.smithi037.stdout:========================================================================================= 2024-03-25T16:27:26.574 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:27:26.574 INFO:teuthology.orchestra.run.smithi037.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:27:26.574 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-25T16:27:26.574 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:27:26.574 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:27:26.574 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:27:26.574 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:27:26.575 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:27:26.576 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:27:26.577 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:27:26.577 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:27:26.577 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:27:26.577 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:27:26.577 INFO:teuthology.orchestra.run.smithi037.stdout:========================================================================================= 2024-03-25T16:27:26.577 INFO:teuthology.orchestra.run.smithi037.stdout:Install 36 Packages 2024-03-25T16:27:26.577 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:27:26.578 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 30 M 2024-03-25T16:27:26.578 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 107 M 2024-03-25T16:27:26.578 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:27:27.627 INFO:teuthology.orchestra.run.smithi037.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 14 kB/s | 6.6 kB 00:00 2024-03-25T16:27:28.120 INFO:teuthology.orchestra.run.smithi117.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 355 kB/s | 1.5 MB 00:04 2024-03-25T16:27:29.920 INFO:teuthology.orchestra.run.smithi117.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 357 kB/s | 2.1 MB 00:06 2024-03-25T16:27:30.604 INFO:teuthology.orchestra.run.smithi117.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 362 kB/s | 247 kB 00:00 2024-03-25T16:27:30.721 INFO:teuthology.orchestra.run.smithi117.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:27:30.762 INFO:teuthology.orchestra.run.smithi117.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:27:30.788 INFO:teuthology.orchestra.run.smithi117.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T16:27:30.813 INFO:teuthology.orchestra.run.smithi117.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.8 MB/s | 47 kB 00:00 2024-03-25T16:27:30.838 INFO:teuthology.orchestra.run.smithi117.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T16:27:30.880 INFO:teuthology.orchestra.run.smithi117.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T16:27:30.956 INFO:teuthology.orchestra.run.smithi117.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 15 MB/s | 1.1 MB 00:00 2024-03-25T16:27:30.997 INFO:teuthology.orchestra.run.smithi117.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:27:31.031 INFO:teuthology.orchestra.run.smithi117.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:27:31.064 INFO:teuthology.orchestra.run.smithi117.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:27:31.190 INFO:teuthology.orchestra.run.smithi117.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 999 kB/s | 125 kB 00:00 2024-03-25T16:27:31.240 INFO:teuthology.orchestra.run.smithi117.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2024-03-25T16:27:31.577 INFO:teuthology.orchestra.run.smithi037.stdout:(2/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 492 kB/s | 2.1 MB 00:04 2024-03-25T16:27:31.790 INFO:teuthology.orchestra.run.smithi117.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 275 kB/s | 151 kB 00:00 2024-03-25T16:27:31.899 INFO:teuthology.orchestra.run.smithi117.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 401 kB/s | 43 kB 00:00 2024-03-25T16:27:32.058 INFO:teuthology.orchestra.run.smithi117.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-03-25T16:27:32.250 INFO:teuthology.orchestra.run.smithi117.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 1.8 MB/s | 358 kB 00:00 2024-03-25T16:27:32.342 INFO:teuthology.orchestra.run.smithi117.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 116 kB/s | 11 kB 00:00 2024-03-25T16:27:32.433 INFO:teuthology.orchestra.run.smithi117.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 193 kB/s | 18 kB 00:00 2024-03-25T16:27:32.526 INFO:teuthology.orchestra.run.smithi117.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 252 kB/s | 23 kB 00:00 2024-03-25T16:27:32.626 INFO:teuthology.orchestra.run.smithi117.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 195 kB/s | 19 kB 00:00 2024-03-25T16:27:32.661 INFO:teuthology.orchestra.run.smithi037.stdout:(3/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 277 kB/s | 1.5 MB 00:05 2024-03-25T16:27:32.717 INFO:teuthology.orchestra.run.smithi117.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 214 kB/s | 20 kB 00:00 2024-03-25T16:27:32.818 INFO:teuthology.orchestra.run.smithi117.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 463 kB/s | 46 kB 00:00 2024-03-25T16:27:32.926 INFO:teuthology.orchestra.run.smithi117.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 729 kB/s | 79 kB 00:00 2024-03-25T16:27:33.052 INFO:teuthology.orchestra.run.smithi117.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-03-25T16:27:33.143 INFO:teuthology.orchestra.run.smithi117.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 179 kB/s | 16 kB 00:00 2024-03-25T16:27:33.252 INFO:teuthology.orchestra.run.smithi117.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 828 kB/s | 90 kB 00:00 2024-03-25T16:27:33.344 INFO:teuthology.orchestra.run.smithi117.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 390 kB/s | 36 kB 00:00 2024-03-25T16:27:33.470 INFO:teuthology.orchestra.run.smithi117.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 1.8 MB/s | 230 kB 00:00 2024-03-25T16:27:33.620 INFO:teuthology.orchestra.run.smithi117.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 2.8 MB/s | 427 kB 00:00 2024-03-25T16:27:33.720 INFO:teuthology.orchestra.run.smithi117.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 200 kB/s | 20 kB 00:00 2024-03-25T16:27:33.844 INFO:teuthology.orchestra.run.smithi037.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 209 kB/s | 247 kB 00:01 2024-03-25T16:27:34.011 INFO:teuthology.orchestra.run.smithi037.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-25T16:27:34.070 INFO:teuthology.orchestra.run.smithi037.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 2.4 MB/s | 144 kB 00:00 2024-03-25T16:27:34.103 INFO:teuthology.orchestra.run.smithi037.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 945 kB/s | 32 kB 00:00 2024-03-25T16:27:34.137 INFO:teuthology.orchestra.run.smithi037.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-25T16:27:34.171 INFO:teuthology.orchestra.run.smithi037.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:27:34.213 INFO:teuthology.orchestra.run.smithi037.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:27:34.280 INFO:teuthology.orchestra.run.smithi037.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:27:34.313 INFO:teuthology.orchestra.run.smithi037.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:27:34.347 INFO:teuthology.orchestra.run.smithi037.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:27:34.380 INFO:teuthology.orchestra.run.smithi037.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:27:34.564 INFO:teuthology.orchestra.run.smithi037.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 682 kB/s | 125 kB 00:00 2024-03-25T16:27:34.631 INFO:teuthology.orchestra.run.smithi037.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 336 kB/s | 22 kB 00:00 2024-03-25T16:27:34.823 INFO:teuthology.orchestra.run.smithi037.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 788 kB/s | 151 kB 00:00 2024-03-25T16:27:34.873 INFO:teuthology.orchestra.run.smithi037.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 870 kB/s | 43 kB 00:00 2024-03-25T16:27:34.974 INFO:teuthology.orchestra.run.smithi037.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-03-25T16:27:35.090 INFO:teuthology.orchestra.run.smithi037.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.0 MB/s | 358 kB 00:00 2024-03-25T16:27:35.141 INFO:teuthology.orchestra.run.smithi037.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-25T16:27:35.199 INFO:teuthology.orchestra.run.smithi037.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 304 kB/s | 18 kB 00:00 2024-03-25T16:27:35.284 INFO:teuthology.orchestra.run.smithi037.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 276 kB/s | 23 kB 00:00 2024-03-25T16:27:35.342 INFO:teuthology.orchestra.run.smithi037.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 337 kB/s | 19 kB 00:00 2024-03-25T16:27:35.475 INFO:teuthology.orchestra.run.smithi037.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 147 kB/s | 20 kB 00:00 2024-03-25T16:27:35.534 INFO:teuthology.orchestra.run.smithi037.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 794 kB/s | 46 kB 00:00 2024-03-25T16:27:35.593 INFO:teuthology.orchestra.run.smithi037.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.3 MB/s | 79 kB 00:00 2024-03-25T16:27:35.660 INFO:teuthology.orchestra.run.smithi037.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.0 MB/s | 272 kB 00:00 2024-03-25T16:27:35.718 INFO:teuthology.orchestra.run.smithi037.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-25T16:27:35.793 INFO:teuthology.orchestra.run.smithi037.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.2 MB/s | 90 kB 00:00 2024-03-25T16:27:36.010 INFO:teuthology.orchestra.run.smithi037.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 165 kB/s | 36 kB 00:00 2024-03-25T16:27:36.077 INFO:teuthology.orchestra.run.smithi037.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-25T16:27:36.178 INFO:teuthology.orchestra.run.smithi037.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-03-25T16:27:36.236 INFO:teuthology.orchestra.run.smithi037.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 343 kB/s | 20 kB 00:00 2024-03-25T16:27:37.645 INFO:teuthology.orchestra.run.smithi117.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 356 kB/s | 4.7 MB 00:13 2024-03-25T16:27:40.994 INFO:teuthology.orchestra.run.smithi037.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 359 kB/s | 4.7 MB 00:13 2024-03-25T16:28:01.727 INFO:teuthology.orchestra.run.smithi037.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 570 kB/s | 17 MB 00:30 2024-03-25T16:28:01.730 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:01.730 INFO:teuthology.orchestra.run.smithi037.stdout:Total 864 kB/s | 30 MB 00:35 2024-03-25T16:28:01.755 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:28:01.800 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:28:01.800 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:28:02.231 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:28:02.231 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:28:03.569 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:28:03.637 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:28:03.683 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:28:03.731 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:28:03.805 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:28:03.855 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:28:03.896 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:28:03.993 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:28:04.069 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:28:04.116 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:28:04.191 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:28:04.243 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:28:04.288 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:28:04.356 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:28:04.418 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:28:04.613 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:28:04.667 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:28:04.706 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:28:04.741 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:28:04.770 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:28:04.848 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:28:04.880 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:28:04.914 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:28:04.954 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:28:04.990 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:28:05.024 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:28:05.125 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:28:05.171 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:28:05.246 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:28:05.388 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:28:05.514 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:28:05.679 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:28:05.709 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:28:05.734 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:28:05.734 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:05.735 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:28:05.735 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:28:05.735 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:28:05.735 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:07.243 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:28:07.270 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:28:07.270 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:07.271 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:28:07.271 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:28:07.271 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:28:07.271 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:07.736 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:28:07.759 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:28:07.759 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:07.759 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:28:07.759 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:28:07.759 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:28:07.759 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:07.978 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:28:08.001 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:28:08.001 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:08.001 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:28:08.001 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:28:08.001 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:28:08.001 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:08.095 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:28:09.630 INFO:teuthology.orchestra.run.smithi117.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 414 kB/s | 17 MB 00:41 2024-03-25T16:28:09.635 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:09.635 INFO:teuthology.orchestra.run.smithi117.stdout:Total 659 kB/s | 30 MB 00:46 2024-03-25T16:28:09.660 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:28:09.700 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:28:09.701 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:28:09.913 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:28:09.913 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:28:09.914 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:28:09.914 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:28:09.914 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:28:09.914 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:28:09.914 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:28:09.914 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:28:09.914 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:28:09.914 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:28:09.914 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:28:09.914 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:28:09.915 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:28:09.916 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:28:09.916 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:28:09.917 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:28:09.917 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:28:09.917 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:28:09.917 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:28:09.917 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:28:09.917 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:28:09.917 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:28:10.105 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:28:10.106 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:28:10.878 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:28:10.879 INFO:teuthology.orchestra.run.smithi037.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:28:10.880 INFO:teuthology.orchestra.run.smithi037.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:10.881 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:28:11.264 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-base 2024-03-25T16:28:11.420 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:28:11.482 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:28:11.514 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:28:11.551 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:28:11.622 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:28:11.667 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:28:11.699 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:28:11.792 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:28:11.858 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:04:00 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:28:11.868 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:28:11.911 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:28:11.957 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:11.982 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:28:12.012 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:28:12.013 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-25T16:28:12.013 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:28:12.036 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:28:12.076 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:28:12.133 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install cephadm 2024-03-25T16:28:12.133 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:28:12.179 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:28:12.369 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:28:12.412 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:28:12.443 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:28:12.477 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:28:12.507 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:28:12.583 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:28:12.617 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:28:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:28:12.691 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:28:12.704 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:04:01 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:28:12.726 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:28:12.769 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:28:12.858 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:28:12.859 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:28:12.860 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:28:12.860 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-25T16:28:12.860 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:28:12.860 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:28:12.860 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:28:12.860 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:12.860 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:28:12.861 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:28:12.861 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T16:28:12.861 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:12.861 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 747 k 2024-03-25T16:28:12.861 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 754 k 2024-03-25T16:28:12.861 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:28:12.900 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:28:12.971 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:28:13.108 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:28:13.233 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:28:13.400 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:28:13.430 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:28:13.455 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:28:13.456 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:13.456 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:28:13.456 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:28:13.456 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:28:13.456 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:14.931 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:28:14.958 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:28:14.959 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:14.959 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:28:14.959 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:28:14.959 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:28:14.959 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:15.374 INFO:teuthology.orchestra.run.smithi037.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 297 kB/s | 747 kB 00:02 2024-03-25T16:28:15.374 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:15.375 INFO:teuthology.orchestra.run.smithi037.stdout:Total 297 kB/s | 747 kB 00:02 2024-03-25T16:28:15.375 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:28:15.381 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:28:15.381 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:28:15.388 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:28:15.388 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:28:15.407 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:28:15.431 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:28:15.431 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:15.431 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:28:15.431 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:28:15.431 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:28:15.431 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:15.463 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:28:15.632 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:28:15.655 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:28:15.655 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:15.655 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:28:15.656 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:28:15.656 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:28:15.656 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:15.721 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:28:16.061 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:16.157 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:16.496 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:16.769 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:16.769 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:16.769 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:28:16.769 INFO:teuthology.orchestra.run.smithi037.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:16.769 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:16.770 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:28:16.978 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:28:17.434 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:28:17.434 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:28:17.434 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:28:17.434 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:28:17.434 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:28:17.434 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:28:17.434 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:28:17.434 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:28:17.434 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:28:17.435 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:28:17.436 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:28:17.557 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:04:06 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:28:17.713 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:28:17.713 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 145 k 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 436 k 2024-03-25T16:28:17.714 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:28:18.166 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:28:18.167 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:28:18.168 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:18.169 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:28:18.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-base 2024-03-25T16:28:18.604 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 163 kB/s | 145 kB 00:00 2024-03-25T16:28:18.605 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:18.605 INFO:teuthology.orchestra.run.smithi037.stdout:Total 163 kB/s | 145 kB 00:00 2024-03-25T16:28:18.605 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:28:18.615 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:28:18.615 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:28:18.672 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:28:18.672 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:28:18.930 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:28:18.989 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:19.013 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:19.013 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:28:19.013 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:19.105 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:04:04 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:28:19.204 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:19.256 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:28:19.257 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-25T16:28:19.257 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:28:19.329 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install cephadm 2024-03-25T16:28:19.886 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:04:04 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:28:19.963 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:19.963 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:19.963 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:28:19.963 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:19.963 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:19.964 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:28:20.033 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:28:20.034 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:28:20.034 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-25T16:28:20.034 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:28:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:28:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:28:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:28:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:28:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T16:28:20.035 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 747 k 2024-03-25T16:28:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 754 k 2024-03-25T16:28:20.035 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:28:20.216 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr 2024-03-25T16:28:20.792 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:04:09 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:28:20.894 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:20.949 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:28:20.950 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-25T16:28:20.950 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:28:21.016 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:28:21.587 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:04:10 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:28:21.743 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:28:21.744 INFO:teuthology.orchestra.run.smithi037.stdout:====================================================================================== 2024-03-25T16:28:21.744 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-25T16:28:21.744 INFO:teuthology.orchestra.run.smithi037.stdout:====================================================================================== 2024-03-25T16:28:21.744 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:28:21.745 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:21.746 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:28:21.746 INFO:teuthology.orchestra.run.smithi037.stdout:====================================================================================== 2024-03-25T16:28:21.746 INFO:teuthology.orchestra.run.smithi037.stdout:Install 12 Packages 2024-03-25T16:28:21.746 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:21.746 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 6.9 M 2024-03-25T16:28:21.746 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 26 M 2024-03-25T16:28:21.747 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:28:22.899 INFO:teuthology.orchestra.run.smithi037.stdout:(1/12): ceph-grafana-dashboards-19.0.0-1439.g5d 53 kB/s | 31 kB 00:00 2024-03-25T16:28:22.933 INFO:teuthology.orchestra.run.smithi037.stdout:(2/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 25 kB/s | 15 kB 00:00 2024-03-25T16:28:23.158 INFO:teuthology.orchestra.run.smithi037.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.0 MB/s | 238 kB 00:00 2024-03-25T16:28:23.192 INFO:teuthology.orchestra.run.smithi037.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.5 MB/s | 1.0 MB 00:00 2024-03-25T16:28:23.834 INFO:teuthology.orchestra.run.smithi037.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 854 kB/s | 548 kB 00:00 2024-03-25T16:28:23.867 INFO:teuthology.orchestra.run.smithi037.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.2 MB/s | 863 kB 00:00 2024-03-25T16:28:23.951 INFO:teuthology.orchestra.run.smithi037.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-03-25T16:28:24.076 INFO:teuthology.orchestra.run.smithi037.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.1 MB/s | 144 kB 00:00 2024-03-25T16:28:24.185 INFO:teuthology.orchestra.run.smithi037.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 284 kB/s | 31 kB 00:00 2024-03-25T16:28:24.302 INFO:teuthology.orchestra.run.smithi037.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 1.6 MB/s | 188 kB 00:00 2024-03-25T16:28:24.381 INFO:teuthology.orchestra.run.smithi117.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 172 kB/s | 747 kB 00:04 2024-03-25T16:28:24.382 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:24.382 INFO:teuthology.orchestra.run.smithi117.stdout:Total 172 kB/s | 747 kB 00:04 2024-03-25T16:28:24.382 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:28:24.389 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:28:24.389 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:28:24.395 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:28:24.395 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:28:24.471 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:28:24.710 INFO:teuthology.orchestra.run.smithi037.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-25T16:28:24.872 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:24.968 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:25.362 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:25.649 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:28:25.649 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:25.650 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:28:25.650 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:25.650 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:25.650 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:28:25.852 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:28:26.436 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:28:26.585 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:28:26.586 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:28:26.586 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T16:28:26.586 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:28:26.587 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:28:26.587 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:28:26.587 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:26.587 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:28:26.587 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:28:26.587 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T16:28:26.587 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:26.587 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 145 k 2024-03-25T16:28:26.587 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 436 k 2024-03-25T16:28:26.587 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:28:28.228 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 89 kB/s | 145 kB 00:01 2024-03-25T16:28:28.229 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:28.229 INFO:teuthology.orchestra.run.smithi117.stdout:Total 88 kB/s | 145 kB 00:01 2024-03-25T16:28:28.229 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:28:28.239 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:28:28.239 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:28:28.296 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:28:28.296 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:28:28.442 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:28:28.513 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:28.537 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:28.537 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:28:28.537 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:28:28.537 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:28.560 INFO:teuthology.orchestra.run.smithi037.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 292 kB/s | 1.8 MB 00:06 2024-03-25T16:28:28.564 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:28.565 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.0 MB/s | 6.9 MB 00:06 2024-03-25T16:28:28.585 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:28:28.605 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:28:28.606 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:28:28.783 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:28:28.784 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:28:29.296 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:28:29.386 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:28:29.430 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:28:29.477 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:28:29.490 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:28:29.490 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:29.490 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:28:29.490 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:28:29.490 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:29.490 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:28:29.514 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:28:29.621 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:28:29.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr 2024-03-25T16:28:29.752 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:28:29.845 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:28:29.903 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:28:29.931 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:28:29.963 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:28:30.216 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:28:30.254 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:28:30.300 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:04:15 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:28:30.398 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:28:30.449 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:28:30.450 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-25T16:28:30.450 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:28:30.518 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:28:31.019 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:28:31.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:28:31.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:28:31.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:28:31.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:28:31.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:28:31.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:28:31.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:28:31.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:28:31.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:28:31.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:28:31.020 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:28:31.075 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:04:16 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:28:31.224 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:28:31.225 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-03-25T16:28:31.225 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:28:31.226 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:28:31.227 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:28:31.227 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:31.227 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:28:31.227 INFO:teuthology.orchestra.run.smithi117.stdout:====================================================================================== 2024-03-25T16:28:31.227 INFO:teuthology.orchestra.run.smithi117.stdout:Install 12 Packages 2024-03-25T16:28:31.227 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:31.227 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 6.9 M 2024-03-25T16:28:31.228 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 26 M 2024-03-25T16:28:31.228 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:28:31.405 INFO:teuthology.orchestra.run.smithi037.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:28:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:28:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:28:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:28:31.406 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:31.406 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:28:31.715 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:28:32.048 INFO:teuthology.orchestra.run.smithi117.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 35 kB/s | 15 kB 00:00 2024-03-25T16:28:32.289 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:04:21 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:28:32.340 INFO:teuthology.orchestra.run.smithi117.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 43 kB/s | 31 kB 00:00 2024-03-25T16:28:32.445 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:28:32.446 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-03-25T16:28:32.446 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-25T16:28:32.446 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-03-25T16:28:32.446 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:28:32.446 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:28:32.446 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout:============================================================================================ 2024-03-25T16:28:32.447 INFO:teuthology.orchestra.run.smithi037.stdout:Install 12 Packages 2024-03-25T16:28:32.448 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:28:32.448 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 37 M 2024-03-25T16:28:32.448 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 209 M 2024-03-25T16:28:32.448 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:28:32.465 INFO:teuthology.orchestra.run.smithi117.stdout:(3/12): protobuf-3.14.0-13.el9.x86_64.rpm 2.4 MB/s | 1.0 MB 00:00 2024-03-25T16:28:32.490 INFO:teuthology.orchestra.run.smithi117.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.6 MB/s | 238 kB 00:00 2024-03-25T16:28:32.741 INFO:teuthology.orchestra.run.smithi117.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 2.1 MB/s | 548 kB 00:00 2024-03-25T16:28:32.833 INFO:teuthology.orchestra.run.smithi117.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 212 kB/s | 19 kB 00:00 2024-03-25T16:28:32.858 INFO:teuthology.orchestra.run.smithi037.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 206 kB/s | 32 kB 00:00 2024-03-25T16:28:32.900 INFO:teuthology.orchestra.run.smithi117.stdout:(7/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.9 MB/s | 863 kB 00:00 2024-03-25T16:28:32.908 INFO:teuthology.orchestra.run.smithi037.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 343 kB/s | 17 kB 00:00 2024-03-25T16:28:32.958 INFO:teuthology.orchestra.run.smithi037.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 685 kB/s | 34 kB 00:00 2024-03-25T16:28:33.050 INFO:teuthology.orchestra.run.smithi117.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 9.4 MB/s | 2.0 MB 00:00 2024-03-25T16:28:33.075 INFO:teuthology.orchestra.run.smithi117.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 824 kB/s | 144 kB 00:00 2024-03-25T16:28:33.092 INFO:teuthology.orchestra.run.smithi037.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 7.7 MB/s | 3.0 MB 00:00 2024-03-25T16:28:33.142 INFO:teuthology.orchestra.run.smithi117.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 335 kB/s | 31 kB 00:00 2024-03-25T16:28:33.167 INFO:teuthology.orchestra.run.smithi117.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-25T16:28:33.309 INFO:teuthology.orchestra.run.smithi037.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 14 MB/s | 4.7 MB 00:00 2024-03-25T16:28:33.401 INFO:teuthology.orchestra.run.smithi037.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 16 MB/s | 4.8 MB 00:00 2024-03-25T16:28:33.426 INFO:teuthology.orchestra.run.smithi037.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.9 MB/s | 217 kB 00:00 2024-03-25T16:28:33.610 INFO:teuthology.orchestra.run.smithi037.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.3 MB/s | 803 kB 00:00 2024-03-25T16:28:33.660 INFO:teuthology.orchestra.run.smithi037.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 195 kB 00:00 2024-03-25T16:28:33.877 INFO:teuthology.orchestra.run.smithi037.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 33 MB/s | 16 MB 00:00 2024-03-25T16:28:34.069 INFO:teuthology.orchestra.run.smithi037.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 603 kB/s | 245 kB 00:00 2024-03-25T16:28:37.508 INFO:teuthology.orchestra.run.smithi117.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 309 kB/s | 1.8 MB 00:05 2024-03-25T16:28:37.513 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:37.513 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.1 MB/s | 6.9 MB 00:06 2024-03-25T16:28:37.533 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:28:37.552 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:28:37.552 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:28:37.730 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:28:37.730 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:28:38.244 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:28:38.342 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:28:38.377 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:28:38.423 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:28:38.454 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:28:38.553 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:28:38.687 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:28:38.773 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:28:38.825 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:28:38.854 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:28:38.877 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:28:39.118 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:28:39.154 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:28:39.914 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:28:39.914 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:28:39.914 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:28:39.914 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:28:39.915 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:28:39.915 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:28:39.915 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:28:39.915 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:28:39.915 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:28:39.915 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:28:39.915 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:28:39.915 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:28:40.271 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:28:40.271 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:40.272 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:28:40.272 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:28:40.272 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:40.272 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:40.272 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:40.272 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:28:40.273 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:28:40.273 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:28:40.273 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:28:40.273 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:28:40.273 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:28:40.273 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:28:40.274 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:28:40.274 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:40.274 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:28:40.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:28:41.095 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:04:26 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:28:41.245 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:28:41.246 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-25T16:28:41.246 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-25T16:28:41.246 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-25T16:28:41.246 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:28:41.246 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:28:41.246 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-25T16:28:41.246 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:28:41.246 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================================ 2024-03-25T16:28:41.247 INFO:teuthology.orchestra.run.smithi117.stdout:Install 12 Packages 2024-03-25T16:28:41.248 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:41.248 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 37 M 2024-03-25T16:28:41.248 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 209 M 2024-03-25T16:28:41.248 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:28:41.773 INFO:teuthology.orchestra.run.smithi117.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 192 kB/s | 32 kB 00:00 2024-03-25T16:28:41.806 INFO:teuthology.orchestra.run.smithi117.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 522 kB/s | 17 kB 00:00 2024-03-25T16:28:41.857 INFO:teuthology.orchestra.run.smithi117.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 703 kB/s | 34 kB 00:00 2024-03-25T16:28:41.907 INFO:teuthology.orchestra.run.smithi117.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-25T16:28:42.249 INFO:teuthology.orchestra.run.smithi117.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 12 MB/s | 4.7 MB 00:00 2024-03-25T16:28:42.307 INFO:teuthology.orchestra.run.smithi117.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 12 MB/s | 4.8 MB 00:00 2024-03-25T16:28:42.333 INFO:teuthology.orchestra.run.smithi117.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-25T16:28:42.508 INFO:teuthology.orchestra.run.smithi117.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-25T16:28:42.558 INFO:teuthology.orchestra.run.smithi117.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 195 kB 00:00 2024-03-25T16:28:42.934 INFO:teuthology.orchestra.run.smithi117.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 25 MB/s | 16 MB 00:00 2024-03-25T16:28:43.684 INFO:teuthology.orchestra.run.smithi117.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 218 kB/s | 245 kB 00:01 2024-03-25T16:28:49.958 INFO:teuthology.orchestra.run.smithi117.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 906 kB/s | 7.4 MB 00:08 2024-03-25T16:28:49.961 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:49.961 INFO:teuthology.orchestra.run.smithi117.stdout:Total 4.3 MB/s | 37 MB 00:08 2024-03-25T16:28:50.005 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:28:50.029 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:28:50.030 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:28:50.388 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:28:50.388 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:28:51.093 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:28:51.155 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:28:51.195 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:28:51.288 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:28:51.323 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:28:51.499 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:28:51.522 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:28:51.606 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:28:52.258 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:28:52.359 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:28:53.217 INFO:teuthology.orchestra.run.smithi037.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 369 kB/s | 7.4 MB 00:20 2024-03-25T16:28:53.220 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:53.220 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.8 MB/s | 37 MB 00:20 2024-03-25T16:28:53.262 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:28:53.288 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:28:53.289 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:28:53.586 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:28:53.664 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:28:53.664 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:28:54.384 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:28:54.449 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:28:54.488 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:28:54.586 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:28:54.617 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:28:54.625 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:28:54.647 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:28:54.811 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:28:54.848 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:28:54.930 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:28:55.628 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:28:55.747 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:28:57.059 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:28:57.182 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:28:57.182 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:28:57.182 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:28:57.182 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:28:57.182 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:28:57.182 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:28:57.183 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:28:57.183 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:28:57.183 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:28:57.183 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:28:57.183 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:28:57.183 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:28:57.628 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:28:57.628 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:57.628 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:28:57.628 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:28:57.628 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:57.629 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:28:57.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:28:58.241 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:28:58.272 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:28:58.550 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:28:58.700 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:28:58.701 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-25T16:28:58.701 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-25T16:28:58.701 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-25T16:28:58.701 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:28:58.701 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:28:58.701 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-25T16:28:58.701 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:28:58.701 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:28:58.701 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-25T16:28:58.702 INFO:teuthology.orchestra.run.smithi117.stdout:Install 13 Packages 2024-03-25T16:28:58.703 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:28:58.703 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 2.0 M 2024-03-25T16:28:58.703 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 26 M 2024-03-25T16:28:58.703 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:28:59.213 INFO:teuthology.orchestra.run.smithi117.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 105 kB/s | 18 kB 00:00 2024-03-25T16:28:59.238 INFO:teuthology.orchestra.run.smithi117.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 123 kB/s | 24 kB 00:00 2024-03-25T16:28:59.322 INFO:teuthology.orchestra.run.smithi117.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 174 kB 00:00 2024-03-25T16:28:59.347 INFO:teuthology.orchestra.run.smithi117.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 132 kB 00:00 2024-03-25T16:28:59.381 INFO:teuthology.orchestra.run.smithi117.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T16:28:59.406 INFO:teuthology.orchestra.run.smithi117.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 730 kB/s | 43 kB 00:00 2024-03-25T16:28:59.582 INFO:teuthology.orchestra.run.smithi117.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 97 kB/s | 52 kB 00:00 2024-03-25T16:28:59.640 INFO:teuthology.orchestra.run.smithi117.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 64 kB/s | 15 kB 00:00 2024-03-25T16:28:59.666 INFO:teuthology.orchestra.run.smithi117.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 113 kB/s | 32 kB 00:00 2024-03-25T16:29:00.016 INFO:teuthology.orchestra.run.smithi117.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 169 kB/s | 59 kB 00:00 2024-03-25T16:29:00.149 INFO:teuthology.orchestra.run.smithi117.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 357 kB/s | 203 kB 00:00 2024-03-25T16:29:00.283 INFO:teuthology.orchestra.run.smithi117.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 336 kB/s | 90 kB 00:00 2024-03-25T16:29:00.541 INFO:teuthology.orchestra.run.smithi117.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.1 MB/s | 1.0 MB 00:00 2024-03-25T16:29:00.545 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:00.545 INFO:teuthology.orchestra.run.smithi117.stdout:Total 1.1 MB/s | 2.0 MB 00:01 2024-03-25T16:29:00.546 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:29:00.560 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:29:00.560 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:29:00.653 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:29:00.655 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:29:01.053 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:29:01.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:29:01.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:29:01.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:29:01.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:29:01.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:29:01.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:29:01.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:29:01.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:29:01.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:29:01.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:29:01.054 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:29:01.165 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:29:01.329 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:29:01.384 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:29:01.439 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:29:01.458 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:29:01.458 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:29:01.459 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:01.460 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:01.476 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:29:01.518 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:29:01.610 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:29:01.720 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:29:01.778 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:29:01.835 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:29:02.411 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:04:51 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:02.501 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:29:02.541 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:29:02.566 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:02.568 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-25T16:29:02.568 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-25T16:29:02.568 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-25T16:29:02.568 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:29:02.568 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:29:02.568 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-25T16:29:02.568 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:29:02.568 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:29:02.568 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:29:02.568 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:29:02.568 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout:Install 13 Packages 2024-03-25T16:29:02.569 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:02.570 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 2.0 M 2024-03-25T16:29:02.570 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 26 M 2024-03-25T16:29:02.570 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:29:02.575 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:29:02.623 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:29:02.645 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:29:02.790 INFO:teuthology.orchestra.run.smithi037.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 140 kB/s | 24 kB 00:00 2024-03-25T16:29:02.832 INFO:teuthology.orchestra.run.smithi037.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 83 kB/s | 18 kB 00:00 2024-03-25T16:29:02.965 INFO:teuthology.orchestra.run.smithi037.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 994 kB/s | 174 kB 00:00 2024-03-25T16:29:02.991 INFO:teuthology.orchestra.run.smithi037.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 835 kB/s | 132 kB 00:00 2024-03-25T16:29:03.058 INFO:teuthology.orchestra.run.smithi037.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 2.2 MB/s | 211 kB 00:00 2024-03-25T16:29:03.074 INFO:teuthology.orchestra.run.smithi037.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 510 kB/s | 43 kB 00:00 2024-03-25T16:29:03.408 INFO:teuthology.orchestra.run.smithi037.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 92 kB/s | 32 kB 00:00 2024-03-25T16:29:03.423 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:29:03.424 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:29:03.425 INFO:teuthology.orchestra.run.smithi037.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 43 kB/s | 15 kB 00:00 2024-03-25T16:29:03.476 INFO:teuthology.orchestra.run.smithi037.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 61 kB/s | 52 kB 00:00 2024-03-25T16:29:03.642 INFO:teuthology.orchestra.run.smithi037.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 866 kB/s | 203 kB 00:00 2024-03-25T16:29:03.782 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:29:03.782 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:03.782 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:29:03.782 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:29:03.782 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:29:03.782 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:29:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:29:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:29:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:29:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:29:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:29:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:29:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:29:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:29:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:29:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:29:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:03.784 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:03.801 INFO:teuthology.orchestra.run.smithi037.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 566 kB/s | 90 kB 00:00 2024-03-25T16:29:03.843 INFO:teuthology.orchestra.run.smithi037.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 161 kB/s | 59 kB 00:00 2024-03-25T16:29:04.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:29:04.268 INFO:teuthology.orchestra.run.smithi037.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.2 MB/s | 1.0 MB 00:00 2024-03-25T16:29:04.273 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:04.273 INFO:teuthology.orchestra.run.smithi037.stdout:Total 1.2 MB/s | 2.0 MB 00:01 2024-03-25T16:29:04.274 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:29:04.290 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:29:04.290 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:29:04.393 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:29:04.393 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:29:04.617 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:04:49 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:04.768 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:04.768 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:29:04.769 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-25T16:29:04.770 INFO:teuthology.orchestra.run.smithi117.stdout:Install 6 Packages 2024-03-25T16:29:04.770 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:04.770 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 6.8 M 2024-03-25T16:29:04.771 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 33 M 2024-03-25T16:29:04.771 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:29:04.911 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:29:05.076 INFO:teuthology.orchestra.run.smithi117.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.3 MB/s | 248 kB 00:00 2024-03-25T16:29:05.097 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:29:05.155 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:29:05.210 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:29:05.218 INFO:teuthology.orchestra.run.smithi117.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T16:29:05.254 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:29:05.295 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:29:05.392 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:29:05.512 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:29:05.543 INFO:teuthology.orchestra.run.smithi117.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 178 kB/s | 58 kB 00:00 2024-03-25T16:29:05.572 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:29:05.635 INFO:teuthology.orchestra.run.smithi117.stdout:(4/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 186 kB/s | 138 kB 00:00 2024-03-25T16:29:05.660 INFO:teuthology.orchestra.run.smithi117.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 436 kB/s | 51 kB 00:00 2024-03-25T16:29:05.693 INFO:teuthology.orchestra.run.smithi117.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 892 kB/s | 548 kB 00:00 2024-03-25T16:29:05.695 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:05.695 INFO:teuthology.orchestra.run.smithi117.stdout:Total 7.3 MB/s | 6.8 MB 00:00 2024-03-25T16:29:05.708 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:29:05.719 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:29:05.719 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:29:05.806 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:29:05.807 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:29:06.100 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:29:06.267 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:29:06.311 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:29:06.345 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:29:06.394 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:29:06.427 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:29:06.476 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:29:06.505 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:29:06.820 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:29:06.908 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:29:06.977 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:29:07.001 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:29:07.287 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:29:07.288 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:29:07.651 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:29:07.652 INFO:teuthology.orchestra.run.smithi037.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:29:07.653 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:29:07.653 INFO:teuthology.orchestra.run.smithi037.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:29:07.653 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:07.653 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:07.810 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:29:07.810 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:29:07.810 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:29:07.810 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:29:07.810 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:29:07.810 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:29:07.935 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:29:08.114 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:29:08.114 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:08.114 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:29:08.115 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:29:08.115 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:29:08.115 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:29:08.115 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:29:08.115 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:29:08.115 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:29:08.115 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:08.115 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:08.343 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-fuse 2024-03-25T16:29:08.506 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:04:57 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:08.663 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:08.664 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-25T16:29:08.664 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-25T16:29:08.664 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-25T16:29:08.664 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout:Install 6 Packages 2024-03-25T16:29:08.665 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:08.666 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 6.8 M 2024-03-25T16:29:08.666 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 33 M 2024-03-25T16:29:08.666 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:29:08.903 INFO:teuthology.orchestra.run.smithi037.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.6 MB/s | 248 kB 00:00 2024-03-25T16:29:08.912 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:04:53 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:09.062 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:09.062 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:09.063 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-25T16:29:09.063 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:09.063 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:29:09.063 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:29:09.063 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-25T16:29:09.063 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:29:09.063 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:09.063 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:29:09.063 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:09.063 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-03-25T16:29:09.063 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:09.064 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 964 k 2024-03-25T16:29:09.064 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.8 M 2024-03-25T16:29:09.064 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:29:09.305 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 326 kB/s | 79 kB 00:00 2024-03-25T16:29:09.578 INFO:teuthology.orchestra.run.smithi037.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 167 kB/s | 138 kB 00:00 2024-03-25T16:29:09.654 INFO:teuthology.orchestra.run.smithi037.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 731 kB/s | 548 kB 00:00 2024-03-25T16:29:09.787 INFO:teuthology.orchestra.run.smithi037.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 473 kB/s | 51 kB 00:00 2024-03-25T16:29:10.021 INFO:teuthology.orchestra.run.smithi037.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 130 kB/s | 58 kB 00:00 2024-03-25T16:29:10.737 INFO:teuthology.orchestra.run.smithi037.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 2.9 MB/s | 5.8 MB 00:01 2024-03-25T16:29:10.742 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:10.742 INFO:teuthology.orchestra.run.smithi037.stdout:Total 3.3 MB/s | 6.8 MB 00:02 2024-03-25T16:29:10.757 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:29:10.770 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:29:10.770 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:29:10.867 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:29:10.867 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:29:11.169 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:29:11.345 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:29:11.407 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:29:11.965 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:29:12.048 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:29:12.116 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:29:12.145 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:29:12.479 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 259 kB/s | 885 kB 00:03 2024-03-25T16:29:12.481 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:12.481 INFO:teuthology.orchestra.run.smithi117.stdout:Total 282 kB/s | 964 kB 00:03 2024-03-25T16:29:12.481 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:29:12.505 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:29:12.505 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:29:12.581 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:29:12.581 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:29:12.825 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:29:12.878 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:29:12.878 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:29:12.878 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:29:12.878 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:29:12.878 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:29:12.878 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:29:12.981 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:29:13.046 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:29:13.189 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:29:13.190 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:13.190 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:29:13.190 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:29:13.190 INFO:teuthology.orchestra.run.smithi037.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:29:13.190 INFO:teuthology.orchestra.run.smithi037.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:29:13.190 INFO:teuthology.orchestra.run.smithi037.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:29:13.190 INFO:teuthology.orchestra.run.smithi037.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:29:13.190 INFO:teuthology.orchestra.run.smithi037.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:29:13.190 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:13.190 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:13.405 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-fuse 2024-03-25T16:29:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:29:13.660 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:29:13.955 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:29:13.955 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:13.955 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:29:13.955 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:29:13.956 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:13.956 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:13.981 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:14.138 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:14.207 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:14.207 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout:Install 2 Packages 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 964 k 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 2.8 M 2024-03-25T16:29:14.208 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:29:14.232 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-volume 2024-03-25T16:29:14.274 INFO:teuthology.orchestra.run.smithi037.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 587 kB/s | 79 kB 00:00 2024-03-25T16:29:14.806 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:04:59 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:14.954 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:14.955 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:14.955 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-03-25T16:29:14.955 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:14.955 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:29:14.955 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:29:14.955 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-25T16:29:14.955 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:29:14.955 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:14.955 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:29:14.955 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:14.956 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-03-25T16:29:14.956 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:14.956 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 332 k 2024-03-25T16:29:14.956 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.5 M 2024-03-25T16:29:14.956 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:29:15.109 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 450 kB/s | 69 kB 00:00 2024-03-25T16:29:15.509 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 477 kB/s | 264 kB 00:00 2024-03-25T16:29:15.510 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:15.511 INFO:teuthology.orchestra.run.smithi117.stdout:Total 599 kB/s | 332 kB 00:00 2024-03-25T16:29:15.511 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:29:15.520 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:29:15.520 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:29:15.554 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:29:15.555 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:29:15.700 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:29:15.832 INFO:teuthology.orchestra.run.smithi037.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 523 kB/s | 885 kB 00:01 2024-03-25T16:29:15.834 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:15.834 INFO:teuthology.orchestra.run.smithi037.stdout:Total 568 kB/s | 964 kB 00:01 2024-03-25T16:29:15.835 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:29:15.860 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:29:15.860 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:29:15.865 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:29:15.895 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:29:15.912 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:29:15.912 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:29:15.912 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:29:15.912 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:15.939 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:29:15.939 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:29:16.121 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:29:16.263 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:29:16.344 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:29:16.663 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:29:16.995 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:29:16.995 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:29:16.997 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:29:16.997 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:16.997 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:29:16.997 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:29:16.997 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:29:16.997 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:16.997 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:17.307 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:29:17.308 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:17.308 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:29:17.308 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:29:17.308 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:17.308 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:17.357 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados-devel 2024-03-25T16:29:17.604 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-volume 2024-03-25T16:29:17.933 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:02 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:18.081 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:18.082 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:18.082 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T16:29:18.082 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:18.082 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:29:18.082 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:29:18.082 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:18.082 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:29:18.083 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:18.083 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T16:29:18.083 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:18.083 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 127 k 2024-03-25T16:29:18.083 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 456 k 2024-03-25T16:29:18.083 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:29:18.183 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:07 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:18.339 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repository Size 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout:Install 2 Packages 2024-03-25T16:29:18.340 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:18.341 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 332 k 2024-03-25T16:29:18.341 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 1.5 M 2024-03-25T16:29:18.341 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:29:18.454 INFO:teuthology.orchestra.run.smithi037.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 613 kB/s | 69 kB 00:00 2024-03-25T16:29:18.690 INFO:teuthology.orchestra.run.smithi117.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 209 kB/s | 127 kB 00:00 2024-03-25T16:29:18.691 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:18.692 INFO:teuthology.orchestra.run.smithi117.stdout:Total 208 kB/s | 127 kB 00:00 2024-03-25T16:29:18.692 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:29:18.697 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:29:18.697 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:29:18.754 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:29:18.754 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:29:18.904 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:29:19.032 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:19.362 INFO:teuthology.orchestra.run.smithi037.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 258 kB/s | 264 kB 00:01 2024-03-25T16:29:19.363 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:19.364 INFO:teuthology.orchestra.run.smithi037.stdout:Total 325 kB/s | 332 kB 00:01 2024-03-25T16:29:19.364 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:29:19.375 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:29:19.375 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:29:19.417 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:29:19.418 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:29:19.501 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:19.576 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:29:19.768 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:29:19.787 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:19.787 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:19.787 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:29:19.787 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:19.787 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:19.787 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:19.806 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:29:19.822 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:29:19.823 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:20.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs2 2024-03-25T16:29:20.543 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:29:20.593 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:05 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:20.693 INFO:teuthology.orchestra.run.smithi117.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:20.747 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:20.748 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-25T16:29:20.748 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:20.815 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs-devel 2024-03-25T16:29:20.875 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:29:20.875 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:20.875 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:29:20.875 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:29:20.875 INFO:teuthology.orchestra.run.smithi037.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:29:20.875 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:20.875 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:21.202 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados-devel 2024-03-25T16:29:21.373 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:06 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:21.521 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:21.522 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:21.522 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T16:29:21.522 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:21.522 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:29:21.522 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:29:21.522 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:21.522 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:29:21.523 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:21.523 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T16:29:21.523 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:21.523 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 31 k 2024-03-25T16:29:21.523 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 140 k 2024-03-25T16:29:21.523 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:29:21.785 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:10 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:21.938 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:21.939 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:21.939 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T16:29:21.939 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:21.939 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:29:21.939 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:29:21.939 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:21.939 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:29:21.939 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:21.940 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T16:29:21.940 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:21.940 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 127 k 2024-03-25T16:29:21.940 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 456 k 2024-03-25T16:29:21.940 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:29:22.102 INFO:teuthology.orchestra.run.smithi117.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 54 kB/s | 31 kB 00:00 2024-03-25T16:29:22.103 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:22.103 INFO:teuthology.orchestra.run.smithi117.stdout:Total 54 kB/s | 31 kB 00:00 2024-03-25T16:29:22.103 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:29:22.109 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:29:22.109 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:29:22.138 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:29:22.139 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:29:22.263 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:29:22.366 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:22.585 INFO:teuthology.orchestra.run.smithi037.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 197 kB/s | 127 kB 00:00 2024-03-25T16:29:22.586 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:22.586 INFO:teuthology.orchestra.run.smithi037.stdout:Total 196 kB/s | 127 kB 00:00 2024-03-25T16:29:22.586 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:29:22.592 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:29:22.592 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:29:22.652 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:29:22.652 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:29:22.753 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:22.804 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:29:22.912 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:23.053 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:23.053 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:23.053 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:29:23.053 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:23.053 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:23.053 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:23.244 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados2 2024-03-25T16:29:23.351 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:23.671 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:23.671 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:23.671 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:29:23.671 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:23.671 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:23.671 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:23.810 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:08 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:23.884 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs2 2024-03-25T16:29:23.906 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:23.957 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:23.958 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-25T16:29:23.958 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:24.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librbd1 2024-03-25T16:29:24.459 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:24.560 INFO:teuthology.orchestra.run.smithi037.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:24.581 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:09 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:24.616 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:24.617 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-25T16:29:24.617 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:24.676 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs-devel 2024-03-25T16:29:24.684 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:24.740 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:24.741 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-25T16:29:24.742 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:24.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rados 2024-03-25T16:29:25.253 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:14 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:25.391 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:10 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:25.409 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:25.410 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:25.410 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T16:29:25.410 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:25.410 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:29:25.410 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:29:25.411 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:25.411 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:29:25.411 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:25.411 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T16:29:25.411 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:25.411 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 31 k 2024-03-25T16:29:25.411 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 140 k 2024-03-25T16:29:25.411 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:29:25.491 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:25.545 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:25.546 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-25T16:29:25.546 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:25.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rgw 2024-03-25T16:29:25.972 INFO:teuthology.orchestra.run.smithi037.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 56 kB/s | 31 kB 00:00 2024-03-25T16:29:25.972 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:25.972 INFO:teuthology.orchestra.run.smithi037.stdout:Total 55 kB/s | 31 kB 00:00 2024-03-25T16:29:25.973 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:29:25.976 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:29:25.976 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:29:26.009 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:29:26.010 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:29:26.133 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:29:26.159 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:26.247 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:26.254 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:26.306 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:26.307 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-25T16:29:26.307 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:26.362 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-cephfs 2024-03-25T16:29:26.675 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:26.925 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:11 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:26.948 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:29:26.948 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:26.949 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:26.949 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:27.021 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:27.071 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:27.072 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-25T16:29:27.072 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:27.106 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados2 2024-03-25T16:29:27.118 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rbd 2024-03-25T16:29:27.678 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:12 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:27.690 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:16 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:27.775 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:27.791 INFO:teuthology.orchestra.run.smithi037.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:27.825 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:27.827 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-25T16:29:27.827 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:27.845 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:27.847 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-25T16:29:27.847 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:27.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-fuse 2024-03-25T16:29:27.902 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librbd1 2024-03-25T16:29:28.444 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:13 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:28.481 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:17 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:28.581 INFO:teuthology.orchestra.run.smithi037.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:28.592 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 85 k 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 231 k 2024-03-25T16:29:28.593 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:29:28.636 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:28.637 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-25T16:29:28.637 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:28.689 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rados 2024-03-25T16:29:29.188 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 144 kB/s | 85 kB 00:00 2024-03-25T16:29:29.189 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:29.189 INFO:teuthology.orchestra.run.smithi117.stdout:Total 143 kB/s | 85 kB 00:00 2024-03-25T16:29:29.189 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:29:29.197 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:29:29.197 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:29:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:29:29.254 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:29:29.280 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:18 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:29.381 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:29.392 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:29:29.436 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:29.437 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-25T16:29:29.437 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:29.486 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:29.490 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rgw 2024-03-25T16:29:29.888 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:30.073 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:30.174 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:30.177 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:30.228 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:30.230 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-25T16:29:30.230 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:30.281 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-cephfs 2024-03-25T16:29:30.432 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-mirror 2024-03-25T16:29:30.883 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:19 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:30.982 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:31.001 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:15 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:31.036 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:31.037 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-25T16:29:31.037 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:31.091 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rbd 2024-03-25T16:29:31.149 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:31.150 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:31.150 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T16:29:31.151 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:31.151 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:29:31.151 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:29:31.151 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:31.151 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:29:31.151 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:31.151 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T16:29:31.151 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:31.151 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 3.1 M 2024-03-25T16:29:31.151 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 13 M 2024-03-25T16:29:31.151 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:29:31.665 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:20 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:31.765 INFO:teuthology.orchestra.run.smithi037.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:29:31.822 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:31.823 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-25T16:29:31.823 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:31.871 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-fuse 2024-03-25T16:29:32.446 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:21 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:32.600 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:32.601 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 85 k 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 231 k 2024-03-25T16:29:32.602 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:29:33.317 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 119 kB/s | 85 kB 00:00 2024-03-25T16:29:33.317 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:33.318 INFO:teuthology.orchestra.run.smithi037.stdout:Total 119 kB/s | 85 kB 00:00 2024-03-25T16:29:33.318 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:29:33.326 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:29:33.326 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:29:33.386 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:29:33.386 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:29:33.525 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:29:33.618 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:34.062 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:34.344 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:34.344 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:34.344 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:29:34.344 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:34.344 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:34.344 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:34.586 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-mirror 2024-03-25T16:29:35.163 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:24 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:35.317 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:35.318 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:35.318 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T16:29:35.318 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:35.318 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:29:35.319 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:29:35.319 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:35.319 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:29:35.319 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:35.319 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T16:29:35.319 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:35.319 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 3.1 M 2024-03-25T16:29:35.319 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 13 M 2024-03-25T16:29:35.319 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:29:37.549 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 494 kB/s | 3.1 MB 00:06 2024-03-25T16:29:37.549 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:37.550 INFO:teuthology.orchestra.run.smithi117.stdout:Total 494 kB/s | 3.1 MB 00:06 2024-03-25T16:29:37.550 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:29:37.561 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:29:37.561 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:29:37.638 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:29:37.639 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:29:38.060 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:29:38.116 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:38.141 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:38.142 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:29:38.142 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:29:38.142 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:29:38.142 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:29:38.142 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:39.222 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:39.223 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:39.223 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:29:39.223 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:39.223 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:39.223 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:39.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-nbd 2024-03-25T16:29:40.014 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:25 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:40.162 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:40.163 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:40.163 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-03-25T16:29:40.163 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:40.164 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:29:40.164 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:29:40.164 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:40.164 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:29:40.164 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:40.164 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T16:29:40.164 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:40.164 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 171 k 2024-03-25T16:29:40.164 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 494 k 2024-03-25T16:29:40.164 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:29:41.011 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 203 kB/s | 171 kB 00:00 2024-03-25T16:29:41.011 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:41.011 INFO:teuthology.orchestra.run.smithi117.stdout:Total 202 kB/s | 171 kB 00:00 2024-03-25T16:29:41.011 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:29:41.019 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:29:41.019 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:29:41.076 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:29:41.076 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:29:41.222 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:29:41.345 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:41.785 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:42.091 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:42.091 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:42.091 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:29:42.092 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:42.092 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:42.092 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:29:42.277 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 454 kB/s | 3.1 MB 00:06 2024-03-25T16:29:42.278 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:42.279 INFO:teuthology.orchestra.run.smithi037.stdout:Total 454 kB/s | 3.1 MB 00:06 2024-03-25T16:29:42.279 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:29:42.291 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:29:42.291 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:29:42.320 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:29:42.371 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:29:42.371 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:29:42.821 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:29:42.894 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:05:27 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:29:42.895 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:42.919 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:42.919 INFO:teuthology.orchestra.run.smithi037.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:29:42.919 INFO:teuthology.orchestra.run.smithi037.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:29:42.919 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:29:42.919 INFO:teuthology.orchestra.run.smithi037.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:29:42.919 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:43.048 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:29:43.049 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:43.049 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T16:29:43.049 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:43.049 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:29:43.049 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:29:43.049 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-03-25T16:29:43.049 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:29:43.049 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:43.049 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:29:43.050 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:29:43.050 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-03-25T16:29:43.050 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:29:43.050 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 91 M 2024-03-25T16:29:43.050 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 498 M 2024-03-25T16:29:43.050 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:29:44.037 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:44.037 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:44.037 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:29:44.037 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:44.037 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:44.037 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:44.256 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-nbd 2024-03-25T16:29:44.839 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:33 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:44.994 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:44.995 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:44.995 INFO:teuthology.orchestra.run.smithi037.stdout: Package Architecture Version Repository Size 2024-03-25T16:29:44.995 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:44.995 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:29:44.995 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:29:44.995 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:44.995 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:29:44.995 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:44.995 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T16:29:44.996 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:44.996 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 171 k 2024-03-25T16:29:44.996 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 494 k 2024-03-25T16:29:44.996 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:29:45.647 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 263 kB/s | 171 kB 00:00 2024-03-25T16:29:45.648 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:29:45.648 INFO:teuthology.orchestra.run.smithi037.stdout:Total 263 kB/s | 171 kB 00:00 2024-03-25T16:29:45.648 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:29:45.657 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:29:45.657 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:29:45.718 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:29:45.718 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:29:45.875 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:29:45.973 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:46.410 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:46.686 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:29:46.686 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:46.686 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:29:46.686 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:46.686 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:46.686 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:29:46.907 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:29:47.483 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:05:36 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:29:47.639 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:29:47.639 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:47.639 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T16:29:47.640 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:47.640 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:29:47.640 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:29:47.640 INFO:teuthology.orchestra.run.smithi037.stdout:Installing dependencies: 2024-03-25T16:29:47.640 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:29:47.640 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:47.640 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:29:47.640 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:29:47.640 INFO:teuthology.orchestra.run.smithi037.stdout:Install 2 Packages 2024-03-25T16:29:47.640 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:29:47.641 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 91 M 2024-03-25T16:29:47.641 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 498 M 2024-03-25T16:29:47.641 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:30:26.805 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 525 kB/s | 22 MB 00:43 2024-03-25T16:31:12.018 INFO:teuthology.orchestra.run.smithi037.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 272 kB/s | 22 MB 01:24 2024-03-25T16:32:26.989 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 426 kB/s | 68 MB 02:43 2024-03-25T16:32:26.990 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:32:26.990 INFO:teuthology.orchestra.run.smithi117.stdout:Total 566 kB/s | 91 MB 02:43 2024-03-25T16:32:26.990 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:32:26.994 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:32:26.995 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:32:27.590 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:32:27.590 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:32:30.687 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:32:37.786 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:32:37.855 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:32:41.216 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:32:41.216 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:32:41.493 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:32:41.493 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:32:41.493 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T16:32:41.493 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:41.494 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:32:41.494 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:32:41.494 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T16:32:41.706 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:32:42.273 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:08:27 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T16:32:42.430 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T16:32:42.431 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:32:42.431 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T16:32:42.431 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:32:42.431 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T16:32:42.431 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:32:42.431 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:32:42.431 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T16:32:42.431 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T16:32:42.431 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T16:32:42.431 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T16:32:42.431 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 416 M 2024-03-25T16:32:42.432 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.7 G 2024-03-25T16:32:42.432 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T16:34:09.683 INFO:teuthology.orchestra.run.smithi037.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 266 kB/s | 68 MB 04:22 2024-03-25T16:34:09.684 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:34:09.684 INFO:teuthology.orchestra.run.smithi037.stdout:Total 354 kB/s | 91 MB 04:22 2024-03-25T16:34:09.684 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:34:09.689 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:34:09.689 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:34:10.297 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:34:10.298 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:34:13.567 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:34:21.105 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:34:21.177 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:34:24.601 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:34:24.601 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:34:24.866 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:34:24.866 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:34:24.866 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:34:24.866 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:24.866 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:34:24.866 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:34:24.867 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:34:25.197 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:34:25.775 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:10:14 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:34:25.929 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:34:25.930 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:34:25.930 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T16:34:25.930 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:34:25.930 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:34:25.930 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:34:25.930 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:34:25.931 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:34:25.931 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:34:25.931 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T16:34:25.931 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:34:25.931 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 416 M 2024-03-25T16:34:25.931 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 1.7 G 2024-03-25T16:34:25.931 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:53:29.934 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 372 kB/s | 416 MB 19:04 2024-03-25T16:53:29.982 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:53:29.982 INFO:teuthology.orchestra.run.smithi037.stdout:Total 372 kB/s | 416 MB 19:04 2024-03-25T16:53:29.982 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:53:29.983 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:53:29.983 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:53:32.525 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:53:32.525 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:54:17.710 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:54:17.824 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:54:24.795 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:54:25.057 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:54:25.057 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:54:25.057 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:54:25.057 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:54:25.057 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:54:25.057 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:54:25.277 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:54:25.853 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:30:14 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:54:26.007 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:54:26.008 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-25T16:54:26.008 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T16:54:26.008 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-25T16:54:26.009 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:54:26.009 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:54:26.009 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:54:26.009 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:54:26.009 INFO:teuthology.orchestra.run.smithi037.stdout:======================================================================================== 2024-03-25T16:54:26.009 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T16:54:26.009 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:54:26.009 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 1.4 M 2024-03-25T16:54:26.009 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 6.4 M 2024-03-25T16:54:26.009 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:54:31.057 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 289 kB/s | 1.4 MB 00:05 2024-03-25T16:54:31.058 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:54:31.058 INFO:teuthology.orchestra.run.smithi037.stdout:Total 289 kB/s | 1.4 MB 00:05 2024-03-25T16:54:31.059 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T16:54:31.063 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T16:54:31.063 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T16:54:31.110 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T16:54:31.110 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T16:54:31.396 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T16:54:31.517 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:54:32.018 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:54:32.314 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:54:32.314 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:54:32.314 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T16:54:32.314 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:54:32.314 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:54:32.314 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T16:54:32.607 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:54:33.182 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:30:22 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T16:54:33.337 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T16:54:33.338 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:54:33.338 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T16:54:33.338 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:54:33.338 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T16:54:33.338 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:54:33.338 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:54:33.338 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T16:54:33.338 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T16:54:33.338 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T16:54:33.338 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T16:54:33.339 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 176 M 2024-03-25T16:54:33.339 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 753 M 2024-03-25T16:54:33.339 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T16:59:14.301 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 268 kB/s | 416 MB 26:31 2024-03-25T16:59:14.302 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:59:14.302 INFO:teuthology.orchestra.run.smithi117.stdout:Total 268 kB/s | 416 MB 26:31 2024-03-25T16:59:14.302 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T16:59:14.305 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T16:59:14.305 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T16:59:16.764 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T16:59:16.765 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T16:59:58.647 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T16:59:58.750 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:00:06.654 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:00:06.912 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:00:06.912 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:00:06.912 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T17:00:06.912 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:00:06.913 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:00:06.913 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:00:07.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:00:07.726 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:35:52 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:00:07.884 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:00:07.885 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-25T17:00:07.885 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T17:00:07.885 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-25T17:00:07.885 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T17:00:07.885 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:00:07.886 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:00:07.886 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T17:00:07.886 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================== 2024-03-25T17:00:07.886 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T17:00:07.886 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:00:07.886 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 1.4 M 2024-03-25T17:00:07.886 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 6.4 M 2024-03-25T17:00:07.886 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T17:00:11.436 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 412 kB/s | 1.4 MB 00:03 2024-03-25T17:00:11.437 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:00:11.437 INFO:teuthology.orchestra.run.smithi117.stdout:Total 411 kB/s | 1.4 MB 00:03 2024-03-25T17:00:11.438 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T17:00:11.442 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T17:00:11.442 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T17:00:11.486 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T17:00:11.486 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T17:00:11.744 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T17:00:11.877 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:00:12.241 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:00:12.506 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:00:12.506 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:00:12.506 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T17:00:12.506 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:00:12.506 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:00:12.506 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:00:12.691 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:00:13.255 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:35:58 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:00:13.402 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:00:13.403 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:00:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T17:00:13.403 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:00:13.403 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T17:00:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:00:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:00:13.403 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T17:00:13.403 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:00:13.403 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T17:00:13.403 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:00:13.404 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 176 M 2024-03-25T17:00:13.404 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 753 M 2024-03-25T17:00:13.404 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T17:04:11.475 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 311 kB/s | 176 MB 09:38 2024-03-25T17:04:11.475 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:04:11.475 INFO:teuthology.orchestra.run.smithi037.stdout:Total 311 kB/s | 176 MB 09:38 2024-03-25T17:04:11.476 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:04:11.479 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:04:11.479 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:04:12.591 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:04:12.592 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:04:31.798 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:04:32.402 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:04:35.790 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:04:36.061 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:04:36.061 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:04:36.062 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:04:36.062 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:04:36.062 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:04:36.062 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:04:36.252 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:04:36.825 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 0:40:25 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:04:36.980 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:04:36.981 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:04:36.981 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:04:36.981 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:04:36.981 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:04:36.981 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:04:36.981 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:04:36.981 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:04:36.981 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:04:36.981 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:04:36.981 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:04:36.981 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 672 M 2024-03-25T17:04:36.982 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 3.1 G 2024-03-25T17:04:36.982 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:08:24.132 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 367 kB/s | 176 MB 08:10 2024-03-25T17:08:24.132 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:08:24.132 INFO:teuthology.orchestra.run.smithi117.stdout:Total 367 kB/s | 176 MB 08:10 2024-03-25T17:08:24.133 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T17:08:24.136 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T17:08:24.136 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T17:08:25.191 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T17:08:25.191 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T17:08:43.116 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T17:08:43.276 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:47.774 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:48.060 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:08:48.061 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:08:48.061 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T17:08:48.061 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:08:48.061 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:08:48.061 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:08:48.258 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:08:48.823 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:44:33 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:08:48.973 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:08:48.974 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:08:48.974 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T17:08:48.974 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:08:48.975 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T17:08:48.975 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:08:48.975 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:08:48.975 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T17:08:48.975 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:08:48.975 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T17:08:48.975 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:08:48.975 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 672 M 2024-03-25T17:08:48.975 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 3.1 G 2024-03-25T17:08:48.975 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T17:28:31.066 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 480 kB/s | 672 MB 23:54 2024-03-25T17:28:31.088 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:28:31.088 INFO:teuthology.orchestra.run.smithi037.stdout:Total 480 kB/s | 672 MB 23:54 2024-03-25T17:28:31.088 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:28:31.089 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:28:31.089 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:28:35.222 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:28:35.223 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:29:50.207 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:29:50.311 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:57.907 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:58.170 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:29:58.171 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:29:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:29:58.171 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:29:58.171 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:29:58.171 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:29:58.493 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:29:59.073 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:05:48 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:29:59.172 INFO:teuthology.orchestra.run.smithi037.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:29:59.227 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:29:59.229 INFO:teuthology.orchestra.run.smithi037.stdout:Nothing to do. 2024-03-25T17:29:59.229 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:29:59.291 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:29:59.868 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:05:48 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:30:00.022 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:30:00.023 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:30:00.023 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:30:00.023 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:30:00.023 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:30:00.023 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:30:00.023 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:30:00.023 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:30:00.023 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:30:00.023 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:30:00.023 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:30:00.024 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 20 M 2024-03-25T17:30:00.024 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 103 M 2024-03-25T17:30:00.024 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:30:49.758 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 406 kB/s | 20 MB 00:49 2024-03-25T17:30:49.759 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:30:49.759 INFO:teuthology.orchestra.run.smithi037.stdout:Total 406 kB/s | 20 MB 00:49 2024-03-25T17:30:49.759 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:30:49.763 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:30:49.763 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:30:49.919 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:30:49.919 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:30:52.275 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:30:52.383 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:53.349 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:53.604 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:30:53.604 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:30:53.604 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:30:53.604 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:30:53.604 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:30:53.604 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:30:53.780 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:30:54.354 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:06:43 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:30:54.508 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 48 M 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 173 M 2024-03-25T17:30:54.509 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:33:29.678 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 315 kB/s | 48 MB 02:35 2024-03-25T17:33:29.695 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:33:29.695 INFO:teuthology.orchestra.run.smithi037.stdout:Total 315 kB/s | 48 MB 02:35 2024-03-25T17:33:29.696 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:33:29.696 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:33:29.696 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:33:30.010 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:33:30.010 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:33:35.034 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:33:35.136 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:36.540 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:36.812 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:33:36.812 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:33:36.812 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:33:36.812 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:33:36.812 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:33:36.812 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:33:36.984 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:33:37.558 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:09:26 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:33:37.713 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:33:37.714 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:33:37.714 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:33:37.714 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:33:37.714 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:33:37.714 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:33:37.715 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:33:37.715 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:33:37.715 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:33:37.715 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:33:37.715 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:33:37.715 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 99 M 2024-03-25T17:33:37.715 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 449 M 2024-03-25T17:33:37.715 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:34:22.226 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 449 kB/s | 672 MB 25:33 2024-03-25T17:34:22.227 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:34:22.227 INFO:teuthology.orchestra.run.smithi117.stdout:Total 449 kB/s | 672 MB 25:33 2024-03-25T17:34:22.227 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T17:34:22.232 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T17:34:22.232 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T17:34:26.201 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T17:34:26.201 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T17:35:37.146 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T17:35:37.406 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:35:44.807 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:35:45.052 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:35:45.053 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:35:45.053 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T17:35:45.053 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:35:45.053 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:35:45.053 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:35:45.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:35:45.900 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:11:30 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:35:45.996 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:35:46.048 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:35:46.049 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-03-25T17:35:46.049 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:35:46.118 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:35:46.684 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:11:31 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:35:46.831 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:35:46.832 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:35:46.832 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T17:35:46.833 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:35:46.833 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T17:35:46.833 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:35:46.833 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:35:46.833 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T17:35:46.833 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:35:46.833 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T17:35:46.833 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:35:46.833 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 20 M 2024-03-25T17:35:46.833 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 103 M 2024-03-25T17:35:46.833 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T17:37:01.427 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 270 kB/s | 20 MB 01:14 2024-03-25T17:37:01.427 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:01.427 INFO:teuthology.orchestra.run.smithi117.stdout:Total 270 kB/s | 20 MB 01:14 2024-03-25T17:37:01.428 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T17:37:01.431 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T17:37:01.431 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T17:37:01.579 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T17:37:01.579 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T17:37:03.798 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T17:37:03.999 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:04.934 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:05.196 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:37:05.196 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:37:05.196 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T17:37:05.197 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:37:05.197 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:37:05.197 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:37:05.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:37:05.931 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:12:50 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:37:06.078 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 48 M 2024-03-25T17:37:06.079 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 173 M 2024-03-25T17:37:06.080 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T17:37:54.078 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 394 kB/s | 99 MB 04:16 2024-03-25T17:37:54.079 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:37:54.079 INFO:teuthology.orchestra.run.smithi037.stdout:Total 394 kB/s | 99 MB 04:16 2024-03-25T17:37:54.079 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:37:54.082 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:37:54.082 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:37:54.720 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:37:54.720 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:38:05.595 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:38:05.703 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:08.573 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:08.883 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:08.883 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:38:08.883 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:38:08.883 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:08.883 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:38:08.883 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:38:09.058 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:38:09.636 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:13:58 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:38:09.791 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:38:09.792 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:38:09.792 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:38:09.792 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:38:09.792 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:38:09.792 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:38:09.792 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:38:09.792 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:38:09.792 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:38:09.793 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:38:09.793 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:38:09.793 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 287 M 2024-03-25T17:38:09.793 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 1.2 G 2024-03-25T17:38:09.793 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:38:27.837 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 598 kB/s | 48 MB 01:21 2024-03-25T17:38:27.839 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:38:27.839 INFO:teuthology.orchestra.run.smithi117.stdout:Total 598 kB/s | 48 MB 01:21 2024-03-25T17:38:27.839 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T17:38:27.842 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T17:38:27.842 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T17:38:28.153 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T17:38:28.153 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T17:38:32.858 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T17:38:32.962 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:34.283 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:34.541 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:34.541 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:38:34.541 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T17:38:34.541 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:34.541 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:38:34.542 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:38:34.723 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:38:35.286 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:14:20 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:38:35.435 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:38:35.436 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:38:35.436 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T17:38:35.436 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:38:35.436 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T17:38:35.436 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:38:35.436 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:38:35.436 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T17:38:35.436 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:38:35.436 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T17:38:35.436 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:38:35.437 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 99 M 2024-03-25T17:38:35.437 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 449 M 2024-03-25T17:38:35.437 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T17:43:30.729 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 342 kB/s | 99 MB 04:55 2024-03-25T17:43:30.730 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:43:30.730 INFO:teuthology.orchestra.run.smithi117.stdout:Total 342 kB/s | 99 MB 04:55 2024-03-25T17:43:30.730 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T17:43:30.733 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T17:43:30.733 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T17:43:31.341 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T17:43:31.341 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T17:43:41.600 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T17:43:41.700 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:44.558 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:44.822 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:44.822 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:43:44.823 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T17:43:44.823 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:43:44.823 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:43:44.823 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:43:45.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:43:45.574 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:19:30 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:43:45.722 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:43:45.723 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:43:45.723 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T17:43:45.723 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:43:45.723 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T17:43:45.723 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:43:45.723 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:43:45.723 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T17:43:45.723 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:43:45.723 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T17:43:45.723 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:43:45.724 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 287 M 2024-03-25T17:43:45.724 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.2 G 2024-03-25T17:43:45.724 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T17:52:44.046 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 336 kB/s | 287 MB 14:34 2024-03-25T17:52:44.046 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:52:44.046 INFO:teuthology.orchestra.run.smithi037.stdout:Total 336 kB/s | 287 MB 14:34 2024-03-25T17:52:44.046 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:52:44.049 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:52:44.049 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:52:45.840 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:52:45.841 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:53:16.948 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:53:17.060 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:24.461 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:24.748 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:53:24.749 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:24.749 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:53:24.749 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:24.749 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:24.749 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:53:24.965 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:53:25.541 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:29:14 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:53:25.695 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:53:25.696 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:53:25.696 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:53:25.696 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:53:25.696 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:53:25.696 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:53:25.696 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:25.696 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:53:25.696 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:53:25.697 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:53:25.697 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:25.697 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 5.9 M 2024-03-25T17:53:25.697 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 28 M 2024-03-25T17:53:25.697 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:53:47.051 INFO:teuthology.orchestra.run.smithi037.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 285 kB/s | 5.9 MB 00:21 2024-03-25T17:53:47.052 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:47.052 INFO:teuthology.orchestra.run.smithi037.stdout:Total 285 kB/s | 5.9 MB 00:21 2024-03-25T17:53:47.052 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:53:47.056 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:53:47.057 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:53:47.132 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:53:47.132 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:53:47.909 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:53:48.011 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:53:48.595 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:53:48.863 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:53:48.863 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:48.863 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:53:48.864 INFO:teuthology.orchestra.run.smithi037.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:48.864 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:48.864 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:53:49.049 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:53:49.622 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:29:38 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:53:49.780 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:53:49.780 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:53:49.781 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:53:49.781 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:53:49.781 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:53:49.781 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:53:49.781 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:49.781 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:53:49.781 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:53:49.781 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:53:49.781 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:49.782 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 109 k 2024-03-25T17:53:49.782 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 440 k 2024-03-25T17:53:49.782 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:53:50.470 INFO:teuthology.orchestra.run.smithi037.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 158 kB/s | 109 kB 00:00 2024-03-25T17:53:50.471 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:53:50.471 INFO:teuthology.orchestra.run.smithi037.stdout:Total 158 kB/s | 109 kB 00:00 2024-03-25T17:53:50.471 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:53:50.475 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:53:50.475 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:53:50.512 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:53:50.513 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:53:50.634 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:53:50.738 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:53:51.134 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:53:51.421 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:53:51.421 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:51.422 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:53:51.422 INFO:teuthology.orchestra.run.smithi037.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:53:51.422 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:51.422 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:53:51.604 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:53:52.178 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:29:41 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:53:52.332 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:53:52.333 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:53:52.333 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:53:52.333 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:53:52.333 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:53:52.333 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:53:52.333 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:52.333 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:53:52.334 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:53:52.334 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:53:52.334 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:53:52.334 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 5.4 M 2024-03-25T17:53:52.334 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 25 M 2024-03-25T17:53:52.334 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:54:06.749 INFO:teuthology.orchestra.run.smithi037.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 380 kB/s | 5.4 MB 00:14 2024-03-25T17:54:06.750 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:54:06.750 INFO:teuthology.orchestra.run.smithi037.stdout:Total 380 kB/s | 5.4 MB 00:14 2024-03-25T17:54:06.751 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:54:06.755 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:54:06.755 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:54:06.826 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:54:06.826 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:54:07.527 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:54:07.625 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:54:08.164 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:54:08.447 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:54:08.447 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:54:08.447 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:54:08.447 INFO:teuthology.orchestra.run.smithi037.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:54:08.448 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:54:08.448 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:54:08.655 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librados2-debuginfo 2024-03-25T17:54:09.230 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:29:58 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:54:09.384 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 44 M 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 253 M 2024-03-25T17:54:09.385 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:55:27.195 INFO:teuthology.orchestra.run.smithi037.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 583 kB/s | 44 MB 01:17 2024-03-25T17:55:27.196 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:55:27.196 INFO:teuthology.orchestra.run.smithi037.stdout:Total 583 kB/s | 44 MB 01:17 2024-03-25T17:55:27.196 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:55:27.199 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:55:27.199 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:55:27.512 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:55:27.513 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:55:32.851 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:55:32.963 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:34.850 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:35.136 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:55:35.136 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:55:35.136 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:55:35.136 INFO:teuthology.orchestra.run.smithi037.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:55:35.136 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:55:35.136 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:55:35.345 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install librbd1-debuginfo 2024-03-25T17:55:35.923 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:31:24 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:55:36.080 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:55:36.081 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:55:36.081 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:55:36.081 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:55:36.082 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:55:36.082 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:55:36.082 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:55:36.082 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:55:36.082 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:55:36.082 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:55:36.082 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:55:36.082 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 52 M 2024-03-25T17:55:36.082 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 308 M 2024-03-25T17:55:36.082 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:56:41.714 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 378 kB/s | 287 MB 12:55 2024-03-25T17:56:41.714 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:56:41.714 INFO:teuthology.orchestra.run.smithi117.stdout:Total 378 kB/s | 287 MB 12:55 2024-03-25T17:56:41.714 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T17:56:41.717 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T17:56:41.717 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T17:56:43.435 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T17:56:43.435 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T17:57:12.988 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T17:57:13.090 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:57:14.262 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:57:14.543 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:57:14.543 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:14.544 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T17:57:14.544 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:14.544 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:14.544 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:57:14.761 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:57:15.342 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:33:00 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:57:15.490 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:57:15.491 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:57:15.491 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T17:57:15.491 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:57:15.491 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T17:57:15.491 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:57:15.491 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:15.492 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T17:57:15.492 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:57:15.492 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T17:57:15.492 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:15.492 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 5.9 M 2024-03-25T17:57:15.492 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 28 M 2024-03-25T17:57:15.492 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T17:57:39.939 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 249 kB/s | 5.9 MB 00:24 2024-03-25T17:57:39.939 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:39.940 INFO:teuthology.orchestra.run.smithi117.stdout:Total 249 kB/s | 5.9 MB 00:24 2024-03-25T17:57:39.940 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T17:57:39.944 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T17:57:39.944 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T17:57:40.016 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T17:57:40.017 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T17:57:40.753 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T17:57:40.862 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:41.484 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:41.759 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:57:41.759 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:41.759 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T17:57:41.759 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:41.759 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:41.760 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:57:41.959 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:57:42.520 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:33:27 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:57:42.668 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:57:42.669 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:57:42.669 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T17:57:42.669 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:57:42.669 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T17:57:42.669 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:57:42.669 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:42.669 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T17:57:42.670 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:57:42.670 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T17:57:42.670 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:42.670 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 109 k 2024-03-25T17:57:42.670 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 440 k 2024-03-25T17:57:42.670 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T17:57:43.874 INFO:teuthology.orchestra.run.smithi117.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 90 kB/s | 109 kB 00:01 2024-03-25T17:57:43.875 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:57:43.876 INFO:teuthology.orchestra.run.smithi117.stdout:Total 90 kB/s | 109 kB 00:01 2024-03-25T17:57:43.876 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T17:57:43.880 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T17:57:43.880 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T17:57:43.917 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T17:57:43.917 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T17:57:44.036 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T17:57:44.149 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:57:44.517 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:57:44.792 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:57:44.792 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:44.792 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T17:57:44.793 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:57:44.793 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:44.793 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:57:45.018 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:57:45.581 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:33:30 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:57:45.727 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 5.4 M 2024-03-25T17:57:45.728 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 25 M 2024-03-25T17:57:45.729 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T17:58:08.357 INFO:teuthology.orchestra.run.smithi117.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 242 kB/s | 5.4 MB 00:22 2024-03-25T17:58:08.358 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:08.358 INFO:teuthology.orchestra.run.smithi117.stdout:Total 242 kB/s | 5.4 MB 00:22 2024-03-25T17:58:08.358 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T17:58:08.362 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T17:58:08.362 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T17:58:08.431 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T17:58:08.432 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T17:58:09.102 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T17:58:09.214 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:09.709 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:09.986 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:58:09.986 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:58:09.986 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T17:58:09.986 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:09.986 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:58:09.986 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T17:58:10.171 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados2-debuginfo 2024-03-25T17:58:10.733 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:33:55 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T17:58:10.883 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T17:58:10.884 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:58:10.884 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T17:58:10.884 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:58:10.884 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T17:58:10.884 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:58:10.884 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:58:10.884 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T17:58:10.884 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T17:58:10.884 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T17:58:10.885 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T17:58:10.885 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 44 M 2024-03-25T17:58:10.885 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 253 M 2024-03-25T17:58:10.885 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T17:58:22.171 INFO:teuthology.orchestra.run.smithi037.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 320 kB/s | 52 MB 02:46 2024-03-25T17:58:22.172 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:22.172 INFO:teuthology.orchestra.run.smithi037.stdout:Total 320 kB/s | 52 MB 02:46 2024-03-25T17:58:22.172 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:58:22.176 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:58:22.176 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:58:22.537 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:58:22.538 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:58:28.889 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:58:28.990 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:58:31.127 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:58:31.410 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T17:58:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:31.411 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:58:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:31.411 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:31.411 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:58:31.620 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T17:58:32.189 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:34:21 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:58:32.346 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:58:32.346 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 615 k 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 2.1 M 2024-03-25T17:58:32.347 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:58:34.666 INFO:teuthology.orchestra.run.smithi037.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 265 kB/s | 615 kB 00:02 2024-03-25T17:58:34.667 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:34.667 INFO:teuthology.orchestra.run.smithi037.stdout:Total 265 kB/s | 615 kB 00:02 2024-03-25T17:58:34.668 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:58:34.671 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:58:34.672 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:58:34.714 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:58:34.715 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:58:34.886 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:58:34.985 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:35.501 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:35.777 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:58:35.777 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:35.777 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:58:35.777 INFO:teuthology.orchestra.run.smithi037.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:35.777 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:35.777 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:58:36.095 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rados-debuginfo 2024-03-25T17:58:36.672 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:34:25 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:58:36.828 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 771 k 2024-03-25T17:58:36.829 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 2.7 M 2024-03-25T17:58:36.830 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:58:39.238 INFO:teuthology.orchestra.run.smithi037.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 320 kB/s | 771 kB 00:02 2024-03-25T17:58:39.238 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:39.238 INFO:teuthology.orchestra.run.smithi037.stdout:Total 320 kB/s | 771 kB 00:02 2024-03-25T17:58:39.238 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:58:39.241 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:58:39.241 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:58:39.285 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:58:39.285 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:58:39.478 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:58:39.579 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:58:40.012 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:58:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T17:58:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:40.284 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:58:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:40.284 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:40.284 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:58:40.518 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T17:58:41.096 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:34:30 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:58:41.252 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:58:41.253 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:41.253 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:58:41.253 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:41.253 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:58:41.253 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T17:58:41.253 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:41.253 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:58:41.253 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:41.253 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:58:41.253 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:41.254 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 1.1 M 2024-03-25T17:58:41.254 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 4.3 M 2024-03-25T17:58:41.254 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:58:47.664 INFO:teuthology.orchestra.run.smithi037.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 180 kB/s | 1.1 MB 00:06 2024-03-25T17:58:47.665 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:47.665 INFO:teuthology.orchestra.run.smithi037.stdout:Total 179 kB/s | 1.1 MB 00:06 2024-03-25T17:58:47.665 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:58:47.669 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:58:47.669 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:58:47.714 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:58:47.714 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:58:47.941 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:58:48.048 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:48.482 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:48.758 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:48.758 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:48.758 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:58:48.758 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:48.759 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:48.759 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:58:48.970 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T17:58:49.545 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:34:38 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:58:49.702 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 337 k 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 1.0 M 2024-03-25T17:58:49.703 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:58:51.367 INFO:teuthology.orchestra.run.smithi037.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 203 kB/s | 337 kB 00:01 2024-03-25T17:58:51.368 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:51.368 INFO:teuthology.orchestra.run.smithi037.stdout:Total 203 kB/s | 337 kB 00:01 2024-03-25T17:58:51.368 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:58:51.372 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:58:51.372 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:58:51.414 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:58:51.414 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:58:51.564 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:58:51.668 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:52.023 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:52.308 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:58:52.309 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:52.309 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:58:52.309 INFO:teuthology.orchestra.run.smithi037.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:52.309 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:52.309 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:58:52.527 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T17:58:53.099 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:34:42 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:58:53.254 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:58:53.255 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:53.255 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:58:53.255 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:53.255 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:58:53.255 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T17:58:53.256 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:53.256 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:58:53.256 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:53.256 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:58:53.256 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:53.256 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 509 k 2024-03-25T17:58:53.256 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 2.5 M 2024-03-25T17:58:53.256 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:58:55.395 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 238 kB/s | 509 kB 00:02 2024-03-25T17:58:55.396 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:55.396 INFO:teuthology.orchestra.run.smithi037.stdout:Total 238 kB/s | 509 kB 00:02 2024-03-25T17:58:55.396 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T17:58:55.400 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T17:58:55.400 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T17:58:55.443 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T17:58:55.443 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T17:58:55.614 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T17:58:55.730 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:56.240 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:56.523 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:56.523 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:56.524 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T17:58:56.524 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:56.524 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:56.524 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T17:58:56.877 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T17:58:57.452 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:34:46 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T17:58:57.607 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T17:58:57.607 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:57.607 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T17:58:57.608 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:57.608 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T17:58:57.608 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T17:58:57.608 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:57.608 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T17:58:57.608 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T17:58:57.608 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T17:58:57.608 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T17:58:57.608 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 52 M 2024-03-25T17:58:57.608 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 298 M 2024-03-25T17:58:57.608 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T17:59:55.098 INFO:teuthology.orchestra.run.smithi117.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 435 kB/s | 44 MB 01:44 2024-03-25T17:59:55.099 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:55.099 INFO:teuthology.orchestra.run.smithi117.stdout:Total 435 kB/s | 44 MB 01:44 2024-03-25T17:59:55.099 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T17:59:55.102 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T17:59:55.102 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T17:59:55.404 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T17:59:55.404 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T18:00:00.452 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T18:00:00.553 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:02.326 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:00:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:00:02.602 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T18:00:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:00:02.602 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:00:02.603 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T18:00:02.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:00:03.394 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:35:48 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T18:00:03.546 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T18:00:03.548 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:00:03.548 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T18:00:03.548 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:00:03.548 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T18:00:03.548 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:00:03.548 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:00:03.548 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T18:00:03.548 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:00:03.548 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T18:00:03.548 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:00:03.548 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 52 M 2024-03-25T18:00:03.549 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 308 M 2024-03-25T18:00:03.549 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T18:01:17.686 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 377 kB/s | 52 MB 02:20 2024-03-25T18:01:17.686 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:17.686 INFO:teuthology.orchestra.run.smithi037.stdout:Total 377 kB/s | 52 MB 02:20 2024-03-25T18:01:17.686 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T18:01:17.689 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T18:01:17.689 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T18:01:18.040 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T18:01:18.040 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T18:01:24.289 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T18:01:24.403 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:01:26.624 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:01:26.914 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:01:26.914 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:01:26.914 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T18:01:26.914 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:26.914 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:01:26.914 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T18:01:27.236 DEBUG:teuthology.orchestra.run.smithi037:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:01:27.814 INFO:teuthology.orchestra.run.smithi037.stdout:Last metadata expiration check: 1:37:16 ago on Mon 25 Mar 2024 04:24:11 PM UTC. 2024-03-25T18:01:27.969 INFO:teuthology.orchestra.run.smithi037.stdout:Dependencies resolved. 2024-03-25T18:01:27.970 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T18:01:27.970 INFO:teuthology.orchestra.run.smithi037.stdout: Package Arch Version Repo Size 2024-03-25T18:01:27.970 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T18:01:27.970 INFO:teuthology.orchestra.run.smithi037.stdout:Installing: 2024-03-25T18:01:27.970 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:01:27.970 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:01:27.970 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction Summary 2024-03-25T18:01:27.970 INFO:teuthology.orchestra.run.smithi037.stdout:================================================================================ 2024-03-25T18:01:27.970 INFO:teuthology.orchestra.run.smithi037.stdout:Install 1 Package 2024-03-25T18:01:27.971 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:01:27.971 INFO:teuthology.orchestra.run.smithi037.stdout:Total download size: 1.3 M 2024-03-25T18:01:27.971 INFO:teuthology.orchestra.run.smithi037.stdout:Installed size: 5.5 M 2024-03-25T18:01:27.971 INFO:teuthology.orchestra.run.smithi037.stdout:Downloading Packages: 2024-03-25T18:01:33.553 INFO:teuthology.orchestra.run.smithi037.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 230 kB/s | 1.3 MB 00:05 2024-03-25T18:01:33.554 INFO:teuthology.orchestra.run.smithi037.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:33.554 INFO:teuthology.orchestra.run.smithi037.stdout:Total 230 kB/s | 1.3 MB 00:05 2024-03-25T18:01:33.554 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction check 2024-03-25T18:01:33.558 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction check succeeded. 2024-03-25T18:01:33.558 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction test 2024-03-25T18:01:33.604 INFO:teuthology.orchestra.run.smithi037.stdout:Transaction test succeeded. 2024-03-25T18:01:33.605 INFO:teuthology.orchestra.run.smithi037.stdout:Running transaction 2024-03-25T18:01:33.857 INFO:teuthology.orchestra.run.smithi037.stdout: Preparing : 1/1 2024-03-25T18:01:33.947 INFO:teuthology.orchestra.run.smithi037.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:34.387 INFO:teuthology.orchestra.run.smithi037.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:34.664 INFO:teuthology.orchestra.run.smithi037.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:01:34.664 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:01:34.664 INFO:teuthology.orchestra.run.smithi037.stdout:Installed: 2024-03-25T18:01:34.664 INFO:teuthology.orchestra.run.smithi037.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:01:34.664 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:01:34.664 INFO:teuthology.orchestra.run.smithi037.stdout:Complete! 2024-03-25T18:01:34.873 DEBUG:teuthology.parallel:result is None 2024-03-25T18:03:19.005 INFO:teuthology.orchestra.run.smithi117.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 272 kB/s | 52 MB 03:15 2024-03-25T18:03:19.005 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:19.005 INFO:teuthology.orchestra.run.smithi117.stdout:Total 272 kB/s | 52 MB 03:15 2024-03-25T18:03:19.006 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T18:03:19.009 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T18:03:19.009 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T18:03:19.351 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T18:03:19.351 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T18:03:25.335 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T18:03:25.510 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:03:27.637 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:03:27.923 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:03:27.923 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:27.923 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T18:03:27.923 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:27.923 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:27.923 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T18:03:28.109 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:03:28.669 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:39:13 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T18:03:28.816 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T18:03:28.817 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:28.817 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T18:03:28.817 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:28.817 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T18:03:28.817 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:03:28.817 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:28.817 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T18:03:28.817 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:28.817 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T18:03:28.817 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:28.818 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 615 k 2024-03-25T18:03:28.818 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.1 M 2024-03-25T18:03:28.818 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T18:03:30.155 INFO:teuthology.orchestra.run.smithi117.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 460 kB/s | 615 kB 00:01 2024-03-25T18:03:30.155 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:30.156 INFO:teuthology.orchestra.run.smithi117.stdout:Total 459 kB/s | 615 kB 00:01 2024-03-25T18:03:30.156 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T18:03:30.160 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T18:03:30.160 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T18:03:30.202 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T18:03:30.202 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T18:03:30.366 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T18:03:30.479 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:30.871 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:31.148 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:03:31.148 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:31.148 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T18:03:31.148 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:31.148 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:31.148 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T18:03:31.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:03:31.912 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:39:16 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T18:03:32.063 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T18:03:32.063 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:32.063 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T18:03:32.063 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:32.063 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T18:03:32.063 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:03:32.063 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:32.064 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T18:03:32.064 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:32.064 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T18:03:32.064 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:32.064 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 771 k 2024-03-25T18:03:32.064 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.7 M 2024-03-25T18:03:32.065 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T18:03:33.074 INFO:teuthology.orchestra.run.smithi117.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 763 kB/s | 771 kB 00:01 2024-03-25T18:03:33.075 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:33.075 INFO:teuthology.orchestra.run.smithi117.stdout:Total 762 kB/s | 771 kB 00:01 2024-03-25T18:03:33.075 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T18:03:33.079 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T18:03:33.079 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T18:03:33.117 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T18:03:33.117 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T18:03:33.295 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T18:03:33.398 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:03:33.798 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:03:34.072 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:03:34.072 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:34.072 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T18:03:34.072 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:34.072 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:34.072 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T18:03:34.283 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:03:34.845 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:39:19 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T18:03:34.993 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T18:03:34.993 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:34.993 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T18:03:34.993 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:34.993 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T18:03:34.993 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:03:34.993 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:34.993 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T18:03:34.993 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:34.994 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T18:03:34.994 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:34.994 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 1.1 M 2024-03-25T18:03:34.994 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 4.3 M 2024-03-25T18:03:34.994 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T18:03:37.052 INFO:teuthology.orchestra.run.smithi117.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 559 kB/s | 1.1 MB 00:02 2024-03-25T18:03:37.053 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:37.054 INFO:teuthology.orchestra.run.smithi117.stdout:Total 559 kB/s | 1.1 MB 00:02 2024-03-25T18:03:37.054 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T18:03:37.058 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T18:03:37.058 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T18:03:37.101 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T18:03:37.101 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T18:03:37.329 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T18:03:37.451 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:37.878 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:38.146 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:38.146 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:38.146 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T18:03:38.146 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:38.146 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:38.146 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T18:03:38.368 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:03:38.932 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:39:23 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T18:03:39.080 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T18:03:39.081 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:39.081 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T18:03:39.081 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:39.081 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T18:03:39.082 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:03:39.082 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:39.082 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T18:03:39.082 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:39.082 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T18:03:39.082 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:39.082 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 337 k 2024-03-25T18:03:39.082 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 1.0 M 2024-03-25T18:03:39.082 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T18:03:40.447 INFO:teuthology.orchestra.run.smithi117.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 247 kB/s | 337 kB 00:01 2024-03-25T18:03:40.448 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:40.448 INFO:teuthology.orchestra.run.smithi117.stdout:Total 247 kB/s | 337 kB 00:01 2024-03-25T18:03:40.448 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T18:03:40.452 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T18:03:40.452 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T18:03:40.491 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T18:03:40.491 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T18:03:40.628 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T18:03:40.729 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:41.094 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:41.353 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:03:41.353 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:41.353 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T18:03:41.353 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:41.353 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:41.353 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T18:03:41.571 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:03:42.130 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:39:27 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T18:03:42.277 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T18:03:42.277 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:42.277 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T18:03:42.277 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:42.277 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T18:03:42.278 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:03:42.278 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:42.278 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T18:03:42.278 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:42.278 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T18:03:42.278 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:42.279 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 509 k 2024-03-25T18:03:42.279 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.5 M 2024-03-25T18:03:42.279 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T18:03:44.391 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 241 kB/s | 509 kB 00:02 2024-03-25T18:03:44.391 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:44.392 INFO:teuthology.orchestra.run.smithi117.stdout:Total 241 kB/s | 509 kB 00:02 2024-03-25T18:03:44.392 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T18:03:44.396 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T18:03:44.396 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T18:03:44.436 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T18:03:44.436 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T18:03:44.600 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T18:03:44.706 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:45.090 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:45.353 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:03:45.353 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:45.353 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T18:03:45.353 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:03:45.353 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:45.353 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T18:03:45.566 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:03:46.129 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:39:31 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T18:03:46.279 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T18:03:46.279 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:46.279 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T18:03:46.279 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:46.280 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T18:03:46.280 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:03:46.280 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:46.280 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T18:03:46.280 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:03:46.280 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T18:03:46.280 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:03:46.280 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 52 M 2024-03-25T18:03:46.280 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 298 M 2024-03-25T18:03:46.281 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T18:06:33.378 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 316 kB/s | 52 MB 02:47 2024-03-25T18:06:33.379 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:33.379 INFO:teuthology.orchestra.run.smithi117.stdout:Total 316 kB/s | 52 MB 02:47 2024-03-25T18:06:33.379 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T18:06:33.381 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T18:06:33.381 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T18:06:33.716 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T18:06:33.716 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T18:06:39.606 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T18:06:39.707 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:06:41.649 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:06:41.922 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:06:41.922 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:41.922 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T18:06:41.922 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:41.922 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:41.922 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T18:06:42.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:06:42.712 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 1:42:27 ago on Mon 25 Mar 2024 04:24:15 PM UTC. 2024-03-25T18:06:42.861 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-03-25T18:06:42.862 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:06:42.862 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-03-25T18:06:42.862 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:06:42.862 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-03-25T18:06:42.862 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:06:42.862 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:42.862 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-03-25T18:06:42.862 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-03-25T18:06:42.863 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-03-25T18:06:42.863 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:42.863 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 1.3 M 2024-03-25T18:06:42.863 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 5.5 M 2024-03-25T18:06:42.863 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-03-25T18:06:47.269 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 291 kB/s | 1.3 MB 00:04 2024-03-25T18:06:47.270 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:06:47.270 INFO:teuthology.orchestra.run.smithi117.stdout:Total 291 kB/s | 1.3 MB 00:04 2024-03-25T18:06:47.270 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-03-25T18:06:47.274 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-03-25T18:06:47.274 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-03-25T18:06:47.318 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-03-25T18:06:47.318 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-03-25T18:06:47.552 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-03-25T18:06:47.652 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:48.066 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:48.337 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:48.338 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-03-25T18:06:48.563 DEBUG:teuthology.parallel:result is None 2024-03-25T18:06:48.564 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:06:48.714 DEBUG:teuthology.orchestra.run.smithi037:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:06:48.739 INFO:teuthology.orchestra.run.smithi037.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:06:48.739 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:06:48.740 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:06:48.741 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:06:48.884 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:06:48.905 INFO:teuthology.orchestra.run.smithi117.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:06:48.905 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:06:48.905 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:06:48.907 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:06:48.907 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:06:48.907 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:06:48.939 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:06:48.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:06:48.973 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:06:48.973 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:06:48.973 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:06:49.010 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:06:49.082 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:06:49.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:06:49.109 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:06:49.176 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:06:49.177 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:06:49.177 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:06:49.208 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:06:49.280 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:06:49.280 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:06:49.307 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:06:49.374 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:06:49.375 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:06:49.375 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:06:49.405 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:06:49.476 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:06:49.476 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:06:49.502 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:06:49.569 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:06:49.649 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:06:49.649 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 777 /var/log/ceph 2024-03-25T18:06:49.652 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /var/log/ceph 2024-03-25T18:06:49.681 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:06:49.681 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:06:49.723 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:06:49.751 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:06:49.751 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:06:49.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:06:49.854 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:06:49.854 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:06:49.855 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621265', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi037.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNdzPNwpNxL8sLSxCBb3qYaQUyuKLhg/+dU6nOMhGbC99bahPbGjo7T9lDbLNGoTdxPzhvw8f2JmpIjH68p7bY=', 'smithi117.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAx0A2u20CfmOD6mD1RM83qmHc3FrWuU9cHfiZY/crmiPZ9xhA8kxQyqM2MmBawk1+PAyifrIOL2XSXvt52zQjE='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:06:49.855 DEBUG:teuthology.orchestra.run.smithi037:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:06:49.865 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:06:49.879 DEBUG:teuthology.orchestra.run.smithi037:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:06:49.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:06:49.948 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:06:49.948 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:06:50.003 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:06:50.004 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_1 2024-03-25T18:06:50.059 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:06:50.059 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:06:50.059 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-25T18:06:50.059 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:06:50.059 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:06:50.059 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-25 16:29:43.168515557 +0000 2024-03-25T18:06:50.059 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-25 16:23:24.681568274 +0000 2024-03-25T18:06:50.059 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-25 16:23:24.681568274 +0000 2024-03-25T18:06:50.060 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-25 16:23:24.681568274 +0000 2024-03-25T18:06:50.060 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:06:50.127 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-25T18:06:50.127 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-25T18:06:50.127 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000207458 s, 2.5 MB/s 2024-03-25T18:06:50.128 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:06:50.186 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_2 2024-03-25T18:06:50.242 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:06:50.242 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:06:50.242 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T18:06:50.242 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:06:50.242 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:06:50.242 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-25 16:29:43.168515557 +0000 2024-03-25T18:06:50.242 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-25 16:23:25.028563601 +0000 2024-03-25T18:06:50.242 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-25 16:23:25.028563601 +0000 2024-03-25T18:06:50.242 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-25 16:23:25.028563601 +0000 2024-03-25T18:06:50.243 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:06:50.309 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-25T18:06:50.309 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-25T18:06:50.309 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000210526 s, 2.4 MB/s 2024-03-25T18:06:50.310 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:06:50.367 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_3 2024-03-25T18:06:50.423 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:06:50.423 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:06:50.423 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T18:06:50.423 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:06:50.423 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:06:50.423 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-25 16:29:43.169515544 +0000 2024-03-25T18:06:50.423 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-25 16:23:25.381558848 +0000 2024-03-25T18:06:50.424 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-25 16:23:25.381558848 +0000 2024-03-25T18:06:50.424 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-25 16:23:25.381558848 +0000 2024-03-25T18:06:50.424 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:06:50.491 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-25T18:06:50.491 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-25T18:06:50.491 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000195537 s, 2.6 MB/s 2024-03-25T18:06:50.492 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:06:50.549 DEBUG:teuthology.orchestra.run.smithi037:> stat /dev/vg_nvme/lv_4 2024-03-25T18:06:50.604 INFO:teuthology.orchestra.run.smithi037.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:06:50.604 INFO:teuthology.orchestra.run.smithi037.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:06:50.605 INFO:teuthology.orchestra.run.smithi037.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T18:06:50.605 INFO:teuthology.orchestra.run.smithi037.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:06:50.605 INFO:teuthology.orchestra.run.smithi037.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:06:50.605 INFO:teuthology.orchestra.run.smithi037.stdout:Access: 2024-03-25 16:29:43.169515544 +0000 2024-03-25T18:06:50.605 INFO:teuthology.orchestra.run.smithi037.stdout:Modify: 2024-03-25 16:23:25.837552708 +0000 2024-03-25T18:06:50.605 INFO:teuthology.orchestra.run.smithi037.stdout:Change: 2024-03-25 16:23:25.837552708 +0000 2024-03-25T18:06:50.605 INFO:teuthology.orchestra.run.smithi037.stdout: Birth: 2024-03-25 16:23:25.837552708 +0000 2024-03-25T18:06:50.605 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:06:50.671 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records in 2024-03-25T18:06:50.671 INFO:teuthology.orchestra.run.smithi037.stderr:1+0 records out 2024-03-25T18:06:50.672 INFO:teuthology.orchestra.run.smithi037.stderr:512 bytes copied, 0.000241957 s, 2.1 MB/s 2024-03-25T18:06:50.673 DEBUG:teuthology.orchestra.run.smithi037:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:06:50.730 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:06:50.730 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:06:50.730 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:06:50.748 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:06:50.748 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-03-25T18:06:50.805 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:06:50.805 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:06:50.805 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-03-25T18:06:50.805 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:06:50.805 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:06:50.805 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-25 16:29:38.394671325 +0000 2024-03-25T18:06:50.805 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-25 16:23:21.426639234 +0000 2024-03-25T18:06:50.805 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-25 16:23:21.426639234 +0000 2024-03-25T18:06:50.805 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-25 16:23:21.426639234 +0000 2024-03-25T18:06:50.806 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:06:50.873 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-25T18:06:50.873 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-25T18:06:50.873 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000216115 s, 2.4 MB/s 2024-03-25T18:06:50.874 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:06:50.932 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-03-25T18:06:50.987 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:06:50.987 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:06:50.987 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-25T18:06:50.987 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:06:50.987 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:06:50.988 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-25 16:29:38.394671325 +0000 2024-03-25T18:06:50.988 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-25 16:23:21.758633042 +0000 2024-03-25T18:06:50.988 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-25 16:23:21.758633042 +0000 2024-03-25T18:06:50.988 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-25 16:23:21.758633042 +0000 2024-03-25T18:06:50.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:06:51.054 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-25T18:06:51.054 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-25T18:06:51.054 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000189076 s, 2.7 MB/s 2024-03-25T18:06:51.055 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:06:51.112 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-03-25T18:06:51.168 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:06:51.168 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:06:51.168 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-25T18:06:51.168 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:06:51.168 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:06:51.168 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-25 16:29:38.394671325 +0000 2024-03-25T18:06:51.168 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-25 16:23:22.106626552 +0000 2024-03-25T18:06:51.169 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-25 16:23:22.106626552 +0000 2024-03-25T18:06:51.169 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-25 16:23:22.106626552 +0000 2024-03-25T18:06:51.169 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:06:51.234 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-25T18:06:51.234 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-25T18:06:51.235 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000212963 s, 2.4 MB/s 2024-03-25T18:06:51.236 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:06:51.291 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-03-25T18:06:51.346 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:06:51.346 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:06:51.346 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:06:51.346 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:06:51.346 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:06:51.347 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-03-25 16:29:38.395671306 +0000 2024-03-25T18:06:51.347 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-03-25 16:23:22.450620135 +0000 2024-03-25T18:06:51.347 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-03-25 16:23:22.450620135 +0000 2024-03-25T18:06:51.347 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-03-25 16:23:22.450620135 +0000 2024-03-25T18:06:51.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:06:51.412 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-03-25T18:06:51.412 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-03-25T18:06:51.412 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000267417 s, 1.9 MB/s 2024-03-25T18:06:51.414 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:06:51.470 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:06:51.470 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:06:51.470 INFO:tasks.ceph:Generating config... 2024-03-25T18:06:51.471 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:06:51.471 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:06:51.471 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:06:51.471 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:06:51.471 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:06:51.471 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:06:51.471 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:06:51.472 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:06:51.472 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:06:51.472 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:06:51.472 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:06:51.473 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:06:51.474 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:06:51.475 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:06:51.475 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:06:51.541 INFO:teuthology.orchestra.run.smithi037.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:06:51.544 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:06:51.590 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:06:51.617 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.37:6789]'), ('mon.c', '[v1:172.21.15.37:6790]'), ('mon.b', '[v1:172.21.15.117:6789]')] 2024-03-25T18:06:51.618 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.37:6789],[v1:172.21.15.37:6790],[v1:172.21.15.117:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:06:51.618 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:06:51.619 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:06:51.672 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.37:6789]' --addv c '[v1:172.21.15.37:6790]' --addv b '[v1:172.21.15.117:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:06:51.758 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:06:51.758 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: generated fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:06:51.758 INFO:teuthology.orchestra.run.smithi037.stdout:setting min_mon_release = quincy 2024-03-25T18:06:51.758 INFO:teuthology.orchestra.run.smithi037.stdout:epoch 0 2024-03-25T18:06:51.758 INFO:teuthology.orchestra.run.smithi037.stdout:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:06:51.758 INFO:teuthology.orchestra.run.smithi037.stdout:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:06:51.758 INFO:teuthology.orchestra.run.smithi037.stdout:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:06:51.758 INFO:teuthology.orchestra.run.smithi037.stdout:min_mon_release 17 (quincy) 2024-03-25T18:06:51.759 INFO:teuthology.orchestra.run.smithi037.stdout:election_strategy: 3 2024-03-25T18:06:51.759 INFO:teuthology.orchestra.run.smithi037.stdout:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:06:51.759 INFO:teuthology.orchestra.run.smithi037.stdout:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:06:51.759 INFO:teuthology.orchestra.run.smithi037.stdout:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:06:51.759 INFO:teuthology.orchestra.run.smithi037.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:06:51.760 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:06:51.775 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID f42897d2-ff81-4b7f-bd91-55865e0be8e5... 2024-03-25T18:06:51.775 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:06:51.817 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout:[global] 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout: chdir = "" 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout: auth supported = cephx 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout: filestore xattr use omap = true 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout: mon clock drift allowed = 1.000 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout: osd crush chooseleaf type = 0 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout: auth debug = true 2024-03-25T18:06:51.865 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on old message = true 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: ms die on bug = true 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: mon pg warn max object skew = 0 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default size = 2 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow primary affinity = true 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd allow pg remap = true 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:06:51.866 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on no sortbitwise = false 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on too few osds = false 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: mon_allow_pool_size_one = true 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: osd default data pool replay window = 5 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: mon allow pool delete = true 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: mon cluster log file level = debug 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: debug asserts on shutdown = true 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: mon health detail to clog = false 2024-03-25T18:06:51.867 INFO:teuthology.orchestra.run.smithi037.stdout: mon host = "[v1:172.21.15.37:6789],[v1:172.21.15.37:6790],[v1:172.21.15.117:6789]" 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: mon client directed command retry = 5 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: mon election default strategy = 3 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: ms bind msgr2 = False 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: ms inject socket failures = 5000 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: ms type = async 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: osd heartbeat grace = 80 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: fsid = f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout:[osd] 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: osd journal size = 100 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub load threshold = 5.0 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: osd scrub max interval = 600 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.868 INFO:teuthology.orchestra.run.smithi037.stdout: osd recover clone overlap = true 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: osd recovery max chunk = 1048576 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug shutdown = true 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug op order = true 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify stray on activate = true 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug trim objects = true 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: osd open classes on start = true 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug pg log writeout = true 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: osd map max advance = 10 2024-03-25T18:06:51.869 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: journal zero on create = true 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: filestore apply finisher threads = 3 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: bdev debug aio = true 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug misdirected ops = true 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: bdev async discard = True 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: bdev enable discard = True 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore allocator = stupid 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore block size = 96636764160 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore fsck on mount = True 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: bluestore zero block detection = True 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluefs = 1 2024-03-25T18:06:51.870 INFO:teuthology.orchestra.run.smithi037.stdout: debug bluestore = 1 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: debug monc = 20 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: debug osd = 20 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: debug rocksdb = 10 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd full ratio = 0.9 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: osd class default list = * 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: osd class load list = * 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: osd client watch timeout = 120 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify cached snaps = True 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: osd debug verify missing on start = True 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: osd fast shutdown = False 2024-03-25T18:06:51.871 INFO:teuthology.orchestra.run.smithi037.stdout: osd max markdown count = 1000 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: osd mclock override recovery settings = True 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: osd objectstore = bluestore 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue = debug_random 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: osd op queue cut off = debug_random 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: osd shutdown pgref assert = True 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr] 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: debug mgr = 20 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: mgr/telemetry/nag = false 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout:[mon] 2024-03-25T18:06:51.872 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: debug mon = 20 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: debug paxos = 20 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: debug auth = 20 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: mon data avail warn = 5 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: mon mgr mkfs grace = 240 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd reporter subtree level = osd 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd prime pg temp = true 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:06:51.873 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: # 1m isn't quite enough 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: mon min osdmap epochs = 50 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: mon osd crush smoke test = False 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune interval = 2 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune min = 15 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: mon scrub interval = 300 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: mon warn on pool no app = False 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: paxos service trim min = 10 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:06:51.874 INFO:teuthology.orchestra.run.smithi037.stdout:[client] 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout: rgw cache enabled = true 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable ops log = true 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout: rgw enable usage log = true 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout: debug monc = 20 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout: debug ms = 1 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout: debug objecter = 20 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout: debug rados = 20 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.a] 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.c] 2024-03-25T18:06:51.875 INFO:teuthology.orchestra.run.smithi037.stdout:[mon.b] 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2024-03-25T18:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2024-03-25T18:06:51.877 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: mon_allow_pool_size_one = true 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = "[v1:172.21.15.37:6789],[v1:172.21.15.37:6790],[v1:172.21.15.117:6789]" 2024-03-25T18:06:51.878 INFO:teuthology.orchestra.run.smithi117.stdout: mon client directed command retry = 5 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: mon election default strategy = 3 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: ms bind msgr2 = False 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: ms inject socket failures = 5000 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: ms type = async 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: osd heartbeat grace = 80 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2024-03-25T18:06:51.879 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug trim objects = true 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.880 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: bdev async discard = True 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: bdev enable discard = True 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore allocator = stupid 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore block size = 96636764160 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore fsck on mount = True 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: bluestore zero block detection = True 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluefs = 1 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: debug bluestore = 1 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 20 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: debug rocksdb = 10 2024-03-25T18:06:51.881 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd full ratio = 0.9 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd class default list = * 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd class load list = * 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd client watch timeout = 120 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify cached snaps = True 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify missing on start = True 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd fast shutdown = False 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd max markdown count = 1000 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd mclock override recovery settings = True 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = bluestore 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue = debug_random 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd op queue cut off = debug_random 2024-03-25T18:06:51.882 INFO:teuthology.orchestra.run.smithi117.stdout: osd shutdown pgref assert = True 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2024-03-25T18:06:51.883 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:06:51.884 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: # 1m isn't quite enough 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: mon min osdmap epochs = 50 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd crush smoke test = False 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune interval = 2 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune min = 15 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: mon scrub interval = 300 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on pool no app = False 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: paxos service trim min = 10 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2024-03-25T18:06:51.885 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2024-03-25T18:06:51.886 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:06:51.886 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:06:51.886 INFO:teuthology.orchestra.run.smithi117.stdout: debug monc = 20 2024-03-25T18:06:51.886 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2024-03-25T18:06:51.886 INFO:teuthology.orchestra.run.smithi117.stdout: debug objecter = 20 2024-03-25T18:06:51.886 INFO:teuthology.orchestra.run.smithi117.stdout: debug rados = 20 2024-03-25T18:06:51.886 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2024-03-25T18:06:51.886 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.c] 2024-03-25T18:06:51.886 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.b] 2024-03-25T18:06:51.887 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:06:51.887 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:06:51.941 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:06:51.941 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:06:51.941 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:06:51.955 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:06:51.955 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:06:52.009 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi037.front.sepia.ceph.com 2024-03-25T18:06:52.009 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:06:52.009 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:06:52.009 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:06:52.119 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:06:52.119 DEBUG:teuthology.orchestra.run.smithi037:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:06:52.133 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi117.front.sepia.ceph.com 2024-03-25T18:06:52.133 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:06:52.133 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:06:52.133 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:06:52.172 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:06:52.172 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:06:52.227 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:06:52.227 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:06:52.227 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:06:52.282 INFO:teuthology.orchestra.run.smithi037.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:06:52.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:06:52.344 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:06:52.346 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:06:52.346 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:06:52.347 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:06:52.388 INFO:teuthology.orchestra.run.smithi037.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:06:52.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:06:52.445 INFO:teuthology.orchestra.run.smithi117.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:06:52.460 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:06:52.460 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi037.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:06:52.460 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:06:52.486 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:06:52.486 INFO:tasks.ceph:role: osd.0 2024-03-25T18:06:52.486 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-25T18:06:52.487 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:06:52.560 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:06:52.560 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:06:52.560 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:06:52.560 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:06:52.561 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:06:52.561 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:06:52.561 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:06:52.561 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:06:52.561 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:06:52.561 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:06:53.068 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-03-25T18:06:53.071 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-03-25T18:06:53.071 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:06:53.194 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:06:53.228 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:06:53.296 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:06:53.296 INFO:tasks.ceph:role: osd.1 2024-03-25T18:06:53.297 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-25T18:06:53.297 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:06:53.371 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:06:53.371 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:06:53.372 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:06:53.372 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:06:53.372 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:06:53.372 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:06:53.372 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:06:53.372 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:06:53.372 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:06:53.372 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:06:53.820 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-03-25T18:06:53.822 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-03-25T18:06:53.822 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:06:53.917 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:06:53.949 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:06:54.018 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:06:54.018 INFO:tasks.ceph:role: osd.2 2024-03-25T18:06:54.018 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-25T18:06:54.018 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:06:54.093 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:06:54.093 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:06:54.094 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:06:54.094 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:06:54.094 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:06:54.094 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:06:54.094 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:06:54.094 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:06:54.094 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:06:54.094 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:06:54.580 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-03-25T18:06:54.582 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-03-25T18:06:54.582 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:06:54.727 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:06:54.763 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:06:54.832 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:06:54.833 INFO:tasks.ceph:role: osd.3 2024-03-25T18:06:54.833 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-25T18:06:54.833 DEBUG:teuthology.orchestra.run.smithi037:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:06:54.908 INFO:teuthology.orchestra.run.smithi037.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:06:54.908 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:06:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:06:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:06:54.909 INFO:teuthology.orchestra.run.smithi037.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:06:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:06:54.909 INFO:teuthology.orchestra.run.smithi037.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:06:54.909 INFO:teuthology.orchestra.run.smithi037.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:06:54.909 INFO:teuthology.orchestra.run.smithi037.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:06:54.909 INFO:teuthology.orchestra.run.smithi037.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:06:55.408 INFO:teuthology.orchestra.run.smithi037.stdout:Discarding blocks...Done. 2024-03-25T18:06:55.410 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi037.front.sepia.ceph.com -o noatime 2024-03-25T18:06:55.410 DEBUG:teuthology.orchestra.run.smithi037:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:06:55.561 DEBUG:teuthology.orchestra.run.smithi037:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:06:55.599 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:06:55.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:55.709+0000 7f0c97e24640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:06:55.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:55.709+0000 7f0c97e24640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:06:55.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:55.709+0000 7f0c97e24640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:06:55.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:55.709+0000 7f0c97e24640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:06:55.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:55.709+0000 7f0c97e24640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:06:55.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:55.710+0000 7f0c97e24640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:06:58.558 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:06:58.590 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:06:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:58.679+0000 7f2f7c993640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:06:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:58.680+0000 7f2f7c993640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:06:58.680 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:58.680+0000 7f2f7c993640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:06:58.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:58.680+0000 7f2f7c993640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:06:58.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:58.680+0000 7f2f7c993640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:06:58.681 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:06:58.680+0000 7f2f7c993640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:07:01.519 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:07:01.550 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:01.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:01.641+0000 7f6c83b47640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:07:01.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:01.641+0000 7f6c83b47640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:07:01.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:01.641+0000 7f6c83b47640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:07:01.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:01.641+0000 7f6c83b47640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:07:01.642 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:01.641+0000 7f6c83b47640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:07:01.643 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:01.642+0000 7f6c83b47640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:07:04.489 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:07:04.520 DEBUG:teuthology.orchestra.run.smithi037:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:04.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:04.609+0000 7f9c00019640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:07:04.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:04.609+0000 7f9c00019640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:07:04.610 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:04.609+0000 7f9c00019640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:07:04.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:04.609+0000 7f9c00019640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:07:04.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:04.609+0000 7f9c00019640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:07:04.611 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:04.610+0000 7f9c00019640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:07:07.456 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:07:07.487 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:07:07.519 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:07.519 INFO:tasks.ceph:role: osd.4 2024-03-25T18:07:07.519 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-25T18:07:07.519 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:07:07.596 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:07.596 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:07.596 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:07.596 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:07.596 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:07.596 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:07.597 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:07.597 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:07.597 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:07.597 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:08.108 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-25T18:07:08.111 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-25T18:07:08.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:07:08.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:07:08.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:07:08.360 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:08.361 INFO:tasks.ceph:role: osd.5 2024-03-25T18:07:08.361 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-25T18:07:08.361 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:07:08.436 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:08.436 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:08.436 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:08.436 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:08.436 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:08.436 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:08.436 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:08.437 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:08.437 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:08.437 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:08.889 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-25T18:07:08.891 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-25T18:07:08.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:07:08.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:07:09.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:07:09.082 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:09.082 INFO:tasks.ceph:role: osd.6 2024-03-25T18:07:09.082 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-25T18:07:09.083 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:07:09.156 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:09.156 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:09.157 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:09.157 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:09.157 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:09.157 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:09.157 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:09.157 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:09.157 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:09.157 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:09.643 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-25T18:07:09.645 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-25T18:07:09.645 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:07:09.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:07:09.806 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:07:09.873 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:07:09.873 INFO:tasks.ceph:role: osd.7 2024-03-25T18:07:09.873 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-25T18:07:09.873 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:07:09.947 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:07:09.947 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:07:09.947 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:07:09.947 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:07:09.947 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:07:09.947 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:07:09.947 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:07:09.947 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:07:09.948 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:07:09.948 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:07:10.455 INFO:teuthology.orchestra.run.smithi117.stdout:Discarding blocks...Done. 2024-03-25T18:07:10.457 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2024-03-25T18:07:10.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:07:10.589 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:07:10.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:10.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:10.751+0000 7ff83b126640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:07:10.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:10.751+0000 7ff83b126640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:07:10.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:10.751+0000 7ff83b126640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:07:10.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:10.751+0000 7ff83b126640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:07:10.752 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:10.751+0000 7ff83b126640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:07:10.753 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:10.752+0000 7ff83b126640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:07:13.592 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:07:13.624 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:13.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:13.715+0000 7fc0de21e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:07:13.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:13.715+0000 7fc0de21e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:07:13.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:13.715+0000 7fc0de21e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:07:13.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:13.715+0000 7fc0de21e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:07:13.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:13.715+0000 7fc0de21e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:07:13.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:13.716+0000 7fc0de21e640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:07:16.565 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:07:16.596 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:16.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:16.687+0000 7fa7bac15640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:07:16.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:16.687+0000 7fa7bac15640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:07:16.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:16.687+0000 7fa7bac15640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:07:16.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:16.687+0000 7fa7bac15640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:07:16.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:16.687+0000 7fa7bac15640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:07:16.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:16.688+0000 7fa7bac15640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:07:19.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:07:19.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:19.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:19.665+0000 7f1767012640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:07:19.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:19.665+0000 7f1767012640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:07:19.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:19.665+0000 7f1767012640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:07:19.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:19.665+0000 7f1767012640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:07:19.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:19.665+0000 7f1767012640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:07:19.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:19.666+0000 7f1767012640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:07:22.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:07:22.538 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:07:22.538 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:07:22.538 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:07:22.569 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:07:22.569 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:07:22.639 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:07:22.640 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:07:22.708 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:07:22.708 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:07:22.777 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:07:22.777 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:07:22.847 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:07:22.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:07:22.878 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:07:22.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:07:22.949 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:07:22.949 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:07:23.019 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:07:23.019 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:07:23.087 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:07:23.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:07:23.155 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:07:23.155 DEBUG:teuthology.orchestra.run.smithi037:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:07:23.172 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:07:23.173 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:07:23.211 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:07:23.211 DEBUG:teuthology.orchestra.run.smithi037:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:07:23.215 DEBUG:teuthology.orchestra.run.smithi117:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:07:23.253 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.y] 2024-03-25T18:07:23.253 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA8vQFmVRTfEBAAMveqUPgmZ6ZaUfnNVrBTTg== 2024-03-25T18:07:23.253 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.0] 2024-03-25T18:07:23.253 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA/vQFmhpJSKhAAMAoT7w45xbIwCEJ8qok1Hw== 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.1] 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBCvQFmW1ySKBAAcLk3aLBpbfdJYP/eXVt1+A== 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.2] 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBFvQFmjBpDJhAA5xoxGX+j8z435UhEPP7Dkw== 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.3] 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBIvQFm5eJhJBAAqUikfYe65CLbOiAGuoTHdA== 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:[mgr.x] 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA8vQFmQXOLFBAAAZxufRLThA+QpKGpVxdSfQ== 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.4] 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBOvQFm9VDcLBAA4walfY44GXGoUwPbC4AIBw== 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.5] 2024-03-25T18:07:23.254 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBRvQFmzEW2KhAABv50aJQ3Nrp6jvBmHE5isQ== 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.6] 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBUvQFmatIHKRAAKRGJ4RuC6E3JWAAI3mq0Fw== 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi037.stdout:[osd.7] 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQBXvQFm7IO6JxAA/+JG8Ct1V6sgg9UBUW98Zw== 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi037.stdout:[client.0] 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA8vQFmMi0vFxAApjVjwPVVJaIblvPAdwWnqw== 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi037.stdout:[client.1] 2024-03-25T18:07:23.255 INFO:teuthology.orchestra.run.smithi037.stdout: key = AQA8vQFmtMKRGhAA8cjOZ8H9vnZUSSjfjWZFrQ== 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.y] 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA8vQFmVRTfEBAAMveqUPgmZ6ZaUfnNVrBTTg== 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.0] 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA/vQFmhpJSKhAAMAoT7w45xbIwCEJ8qok1Hw== 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.1] 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBCvQFmW1ySKBAAcLk3aLBpbfdJYP/eXVt1+A== 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.2] 2024-03-25T18:07:23.276 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBFvQFmjBpDJhAA5xoxGX+j8z435UhEPP7Dkw== 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.3] 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBIvQFm5eJhJBAAqUikfYe65CLbOiAGuoTHdA== 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.x] 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA8vQFmQXOLFBAAAZxufRLThA+QpKGpVxdSfQ== 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.4] 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBOvQFm9VDcLBAA4walfY44GXGoUwPbC4AIBw== 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.5] 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBRvQFmzEW2KhAABv50aJQ3Nrp6jvBmHE5isQ== 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.6] 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBUvQFmatIHKRAAKRGJ4RuC6E3JWAAI3mq0Fw== 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.7] 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBXvQFm7IO6JxAA/+JG8Ct1V6sgg9UBUW98Zw== 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout:[client.0] 2024-03-25T18:07:23.277 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA8vQFmMi0vFxAApjVjwPVVJaIblvPAdwWnqw== 2024-03-25T18:07:23.278 INFO:teuthology.orchestra.run.smithi117.stdout:[client.1] 2024-03-25T18:07:23.278 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA8vQFmtMKRGhAA8cjOZ8H9vnZUSSjfjWZFrQ== 2024-03-25T18:07:23.278 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:07:23.299 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:07:23.371 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.373 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.425 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.427 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.478 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.532 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.586 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:07:23.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:07:23.639 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.641 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.692 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.746 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.748 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.836 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.837 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:07:23.889 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:07:23.892 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:07:23.946 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:07:23.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:07:24.002 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:07:24.002 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:07:24.028 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:07:24.123 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:07:24.149 DEBUG:teuthology.orchestra.run.smithi037:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:07:24.216 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:07:24.312 DEBUG:teuthology.orchestra.run.smithi037:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:07:24.338 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:07:24.366 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:07:24.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:07:24.488 DEBUG:teuthology.orchestra.run.smithi037:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:24.490 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:07:24.542 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:07:24.543 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:07:24.543 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:07:24.543 DEBUG:teuthology.orchestra.run.smithi037:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:07:24.545 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:07:24.545 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:07:24.545 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:07:24.545 DEBUG:teuthology.orchestra.run.smithi037:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:07:24.547 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:07:24.547 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:07:24.547 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:07:24.547 DEBUG:teuthology.orchestra.run.smithi117:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:07:24.585 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:07:24.585 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:07:24.585 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:07:24.585 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:07:24.587 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:07:24.588 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:07:24.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:07:24.590 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:07:24.590 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:07:24.590 DEBUG:teuthology.orchestra.run.smithi037:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:07:24.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.691+0000 7f28ddf54640 10 monclient: get_monmap_and_config 2024-03-25T18:07:24.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.691+0000 7f28ddf54640 10 monclient: build_initial_monmap 2024-03-25T18:07:24.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.691+0000 7f28ddf54640 10 monclient: monmap: 2024-03-25T18:07:24.693 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-03-25T18:07:24.693 INFO:teuthology.orchestra.run.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:24.693 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-25T18:07:24.693054+0000 2024-03-25T18:07:24.693 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-25T18:07:24.693054+0000 2024-03-25T18:07:24.693 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:24.693 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:24.693 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:24.693 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:24.693 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:24.693 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.691+0000 7f28ddf54640 1 Processor -- start 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.691+0000 7f28ddf54640 1 -- start start 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.691+0000 7f28ddf54640 10 monclient: init 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28ddf54640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28ddf54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28ddf54640 10 monclient(hunting): picked mon.noname-a con 0x7f28d80ad1b0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28ddf54640 10 monclient(hunting): picked mon.noname-b con 0x7f28d80ad590 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28ddf54640 10 monclient(hunting): picked mon.noname-c con 0x7f28d80aaa10 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28ddf54640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80aee40 con 0x7f28d80ad1b0 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28ddf54640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80ae980 con 0x7f28d80aaa10 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28ddf54640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80aeb10 con 0x7f28d80ad590 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28ddf54640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:24.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28ddf54640 10 monclient(hunting): authenticate will time out at 6799.945896s 2024-03-25T18:07:24.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28d8072540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:24.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28ceffd640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28d80ad590 legacy=0x7f28d81401f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:24.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.692+0000 7f28d7fff640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28d80717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:24.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.893+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28d8072540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.893+0000 7f28ceffd640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28d80ad590 legacy=0x7f28d81401f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:24.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:24.893+0000 7f28d7fff640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28d80717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:25.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:25.294+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28d8072540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:25.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:25.294+0000 7f28ceffd640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28d80ad590 legacy=0x7f28d81401f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:25.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:25.294+0000 7f28d7fff640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28d80717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:26.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:26.095+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28d8072540 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:26.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:26.095+0000 7f28ceffd640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28d80ad590 legacy=0x7f28d81401f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:26.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:26.095+0000 7f28d7fff640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28d80717f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:27.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 10 monclient(hunting): tick 2024-03-25T18:07:27.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:07:27.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:07:27.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:07:27.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28d80ad590 legacy=0x7f28d81401f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:27.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28d80717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:27.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28d8072540 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:27.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:07:27.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f28b8003680 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:27.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f28d80ad1b0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:27.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f28d80aaa10 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:27.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d814fb10 con 0x7f28d80ad1b0 2024-03-25T18:07:27.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80aee40 con 0x7f28d80aaa10 2024-03-25T18:07:27.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80ae980 con 0x7f28b8003680 2024-03-25T18:07:27.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d5ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:27.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b8005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:27.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28ceffd640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b80092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:27.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.692+0000 7f28d7fff640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:27.894 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.893+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b8005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:27.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.893+0000 7f28ceffd640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b80092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:27.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:27.893+0000 7f28d7fff640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:28.294+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b8005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:28.294+0000 7f28ceffd640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b80092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:28.294+0000 7f28d7fff640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:29.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:29.095+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b8005ab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:29.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:29.095+0000 7f28ceffd640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b80092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:29.096 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:29.095+0000 7f28d7fff640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:30.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 10 monclient(hunting): tick 2024-03-25T18:07:30.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:07:30.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:07:30.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:07:30.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b8005ab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:30.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:30.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b80092b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:30.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:30.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f28b8003680 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:30.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f28d80aaa10 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:30.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f28d80ad1b0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:30.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80aeb10 con 0x7f28d80ad1b0 2024-03-25T18:07:30.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d814fb10 con 0x7f28d80aaa10 2024-03-25T18:07:30.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80aee40 con 0x7f28b8003680 2024-03-25T18:07:30.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.693+0000 7f28d5ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:30.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.694+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b80103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:30.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.694+0000 7f28d7fff640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b80092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:30.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.694+0000 7f28ceffd640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:30.895 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.895+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b80103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:30.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.895+0000 7f28d7fff640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b80092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:30.896 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:30.895+0000 7f28ceffd640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:31.296 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:31.296+0000 7f28d7fff640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b80092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:31.296+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b80103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:31.297 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:31.296+0000 7f28ceffd640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:31.536 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:00:06.850 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:31.841 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:00:07.155 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:32.061 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:00:07.407 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:32.061 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:00:07.407 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:32.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:32.097+0000 7f28d7fff640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b80092b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6789/0 2024-03-25T18:07:32.097 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:32.097+0000 7f28d77fe640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b80103e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.37:6790/0 2024-03-25T18:07:32.098 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:32.097+0000 7f28ceffd640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.117:6789/0 2024-03-25T18:07:32.388 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:00:07.733 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:32.388 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:00:07.734 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:07:33.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 10 monclient(hunting): tick 2024-03-25T18:07:33.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:07:33.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:07:33.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:07:33.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 1 -- >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b80103e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:33.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 1 -- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:33.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 1 -- >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b80092b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:07:33.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:07:33.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f28b8003680 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:33.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f28d80ad1b0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:33.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f28d80aaa10 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:33.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80ae980 con 0x7f28d80ad1b0 2024-03-25T18:07:33.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80aeb10 con 0x7f28d80aaa10 2024-03-25T18:07:33.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d814fb10 con 0x7f28b8003680 2024-03-25T18:07:33.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.694+0000 7f28d5ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:33.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.748+0000 7f28d7fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 0x7f28b800cab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:38816/0 (socket says 172.21.15.37:38816) 2024-03-25T18:07:33.749 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:33.748+0000 7f28d7fff640 1 -- 172.21.15.37:0/3861442172 learned_addr learned my addr 172.21.15.37:0/3861442172 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:07:34.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:34.082+0000 7f28d77fe640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b80103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T18:07:34.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:34.082+0000 7f28d77fe640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b80103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:07:34.083 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:34.082+0000 7f28d77fe640 1 --1- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 0x7f28b80103a0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:07:34.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:34.089+0000 7f28ceffd640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b8010ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T18:07:34.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:34.089+0000 7f28ceffd640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b8010ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:07:34.090 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:34.089+0000 7f28ceffd640 1 --1- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 0x7f28b8010ab0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:07:36.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 10 monclient(hunting): tick 2024-03-25T18:07:36.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:07:36.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:07:36.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:07:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6790/0 conn(0x7f28b8003680 legacy=0x7f28b80103a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6789/0 conn(0x7f28d80ad1b0 legacy=0x7f28b8010ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f28b8005b00 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f28b8012470 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f28d80aaa10 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d80aee40 con 0x7f28b8005b00 2024-03-25T18:07:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b8010bb0 con 0x7f28d80aaa10 2024-03-25T18:07:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b8010d40 con 0x7f28b8012470 2024-03-25T18:07:36.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:36.694+0000 7f28d5ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 10 monclient(hunting): tick 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6790/0 conn(0x7f28b8012470 legacy=0x7f28b8014860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b8010400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6789/0 conn(0x7f28b8005b00 legacy=0x7f28b80092b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f28b80060f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f28d80aaa10 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f28b8005b00 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:39.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b8010f20 con 0x7f28b8005b00 2024-03-25T18:07:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b8006d40 con 0x7f28d80aaa10 2024-03-25T18:07:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b8006f70 con 0x7f28b80060f0 2024-03-25T18:07:39.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:39.694+0000 7f28d5ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 10 monclient(hunting): tick 2024-03-25T18:07:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:07:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 1 monclient(hunting): continuing hunt 2024-03-25T18:07:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:07:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6790/0 conn(0x7f28b80060f0 legacy=0x7f28b8006530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b8010400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6789/0 conn(0x7f28b8005b00 legacy=0x7f28b800cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:42.696 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-a con 0x7f28b80094f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-b con 0x7f28b80060f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 10 monclient(hunting): picked mon.noname-c con 0x7f28d80aaa10 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b8009360 con 0x7f28b80094f0 2024-03-25T18:07:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b8009aa0 con 0x7f28d80aaa10 2024-03-25T18:07:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28b800cb60 con 0x7f28b80060f0 2024-03-25T18:07:42.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.695+0000 7f28d5ffb640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:42.946 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.945+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 720889843 0 0) 0x7f28b8009360 con 0x7f28b80094f0 2024-03-25T18:07:42.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.945+0000 7f28d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:42.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.945+0000 7f28d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:42.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.945+0000 7f28d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:42.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.945+0000 7f28d6ffd640 10 monclient(hunting): my global_id is 4104 2024-03-25T18:07:42.947 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.945+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28c4003520 con 0x7f28b80094f0 2024-03-25T18:07:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.979+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2764083558 0 0) 0x7f28b8009aa0 con 0x7f28d80aaa10 2024-03-25T18:07:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.979+0000 7f28d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.979+0000 7f28d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.979+0000 7f28d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:42.980 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.979+0000 7f28d6ffd640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:07:42.981 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.979+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b8009360 con 0x7f28d80aaa10 2024-03-25T18:07:42.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.993+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547541775 0 0) 0x7f28b800cb60 con 0x7f28b80060f0 2024-03-25T18:07:42.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.993+0000 7f28d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:42.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.993+0000 7f28d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:42.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.993+0000 7f28d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:42.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.993+0000 7f28d6ffd640 10 monclient(hunting): my global_id is 4103 2024-03-25T18:07:42.994 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:42.993+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b8009aa0 con 0x7f28b80060f0 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3105861886 0 0) 0x7f28c4003520 con 0x7f28b80094f0 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6790/0 conn(0x7f28b80060f0 legacy=0x7f28b8006530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28b8010400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 1 monclient: found mon.noname-a 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28b800cd90 con 0x7f28b80094f0 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:43.005 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f28c8004150 con 0x7f28b80094f0 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28ddf54640 5 monclient: authenticate success, global_id 4104 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28ddf54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.003+0000 7f28d6ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.004+0000 7f28d6ffd640 10 monclient: dump: 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:43.006 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:43.007 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:43.007 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.004+0000 7f28ddf54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:43.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.035+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28c80042b0 con 0x7f28b80094f0 2024-03-25T18:07:43.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.035+0000 7f28d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:43.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.035+0000 7f28d6ffd640 1 -- 172.21.15.37:0/3861442172 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f28c8004700 con 0x7f28b80094f0 2024-03-25T18:07:43.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.035+0000 7f28d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:43.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.035+0000 7f28d6ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:43.036 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.035+0000 7f28d6ffd640 10 monclient: dump: 2024-03-25T18:07:43.036 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-25T18:07:43.036 INFO:teuthology.orchestra.run.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:43.036 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:43.037 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.035+0000 7f28ddf54640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:43.037 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.035+0000 7f28ddf54640 4 set_mon_vals no callback set 2024-03-25T18:07:43.038 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.036+0000 7f28ddf54640 10 monclient: shutdown 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.036+0000 7f28ddf54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.036+0000 7f28ddf54640 1 -- 172.21.15.37:0/3861442172 >> v1:172.21.15.37:6789/0 conn(0x7f28b80094f0 legacy=0x7f28b8012850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.036+0000 7f28ddf54640 1 -- 172.21.15.37:0/3861442172 shutdown_connections 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.036+0000 7f28ddf54640 1 -- 172.21.15.37:0/3861442172 >> 172.21.15.37:0/3861442172 conn(0x7f28d806b560 msgr2=0x7f28d80b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.036+0000 7f28ddf54640 1 -- 172.21.15.37:0/3861442172 shutdown_connections 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.036+0000 7f28ddf54640 1 -- 172.21.15.37:0/3861442172 wait complete. 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.037+0000 7f28ddf54640 10 monclient: build_initial_monmap 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.037+0000 7f28ddf54640 10 monclient: monmap: 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-25T18:07:43.039 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:43.040 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:43.040 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:43.040 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:43.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.038+0000 7f28ddf54640 1 librados: starting msgr at 2024-03-25T18:07:43.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.038+0000 7f28ddf54640 1 librados: starting objecter 2024-03-25T18:07:43.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.038+0000 7f28ddf54640 1 Processor -- start 2024-03-25T18:07:43.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.038+0000 7f28ddf54640 1 -- start start 2024-03-25T18:07:43.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.038+0000 7f28ddf54640 1 librados: setting wanted keys 2024-03-25T18:07:43.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.038+0000 7f28ddf54640 1 librados: calling monclient init 2024-03-25T18:07:43.040 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.038+0000 7f28ddf54640 10 monclient: init 2024-03-25T18:07:43.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.039+0000 7f28ddf54640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.039+0000 7f28ddf54640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.039+0000 7f28ddf54640 10 monclient(hunting): picked mon.noname-a con 0x7f28b80060f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.039+0000 7f28ddf54640 10 monclient(hunting): picked mon.noname-c con 0x7f28b80094f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.039+0000 7f28ddf54640 10 monclient(hunting): picked mon.noname-b con 0x7f28d80aaa10 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.039+0000 7f28ddf54640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d8130f70 con 0x7f28b80060f0 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.039+0000 7f28ddf54640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d813c440 con 0x7f28d80aaa10 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.039+0000 7f28ddf54640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28d813c720 con 0x7f28b80094f0 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.039+0000 7f28ddf54640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.039+0000 7f28ddf54640 10 monclient(hunting): authenticate will time out at 6818.291771s 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.039+0000 7f28ceffd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f28b80094f0 0x7f28d813ad60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:50684/0 (socket says 172.21.15.37:50684) 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.040+0000 7f28ceffd640 1 -- 172.21.15.37:0/884286135 learned_addr learned my addr 172.21.15.37:0/884286135 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.040+0000 7f28d4ff9640 10 client.?.objecter ms_handle_connect 0x7f28b80094f0 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.040+0000 7f28d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.040+0000 7f28d4ff9640 10 client.?.objecter ms_handle_connect 0x7f28d80aaa10 2024-03-25T18:07:43.042 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.040+0000 7f28d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:43.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.090+0000 7f28d4ff9640 10 client.?.objecter ms_handle_connect 0x7f28b80060f0 2024-03-25T18:07:43.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.090+0000 7f28d4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:43.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.103+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1943147640 0 0) 0x7f28d8130f70 con 0x7f28b80060f0 2024-03-25T18:07:43.103 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.103+0000 7f28d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:43.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.103+0000 7f28d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:43.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.103+0000 7f28d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:43.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.103+0000 7f28d4ff9640 10 monclient(hunting): my global_id is 4107 2024-03-25T18:07:43.104 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.103+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b800cb60 con 0x7f28b80060f0 2024-03-25T18:07:43.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.120+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055103958 0 0) 0x7f28d813c440 con 0x7f28d80aaa10 2024-03-25T18:07:43.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.120+0000 7f28d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:43.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.120+0000 7f28d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:43.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.120+0000 7f28d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:43.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.120+0000 7f28d4ff9640 10 monclient(hunting): my global_id is 4108 2024-03-25T18:07:43.121 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.120+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28d8130f70 con 0x7f28d80aaa10 2024-03-25T18:07:43.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1927835065 0 0) 0x7f28b800cb60 con 0x7f28b80060f0 2024-03-25T18:07:43.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f28d813c440 con 0x7f28b80060f0 2024-03-25T18:07:43.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f28c80045c0 con 0x7f28b80060f0 2024-03-25T18:07:43.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:43.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:43.126 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 10 monclient(hunting): dump: 2024-03-25T18:07:43.126 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-25T18:07:43.126 INFO:teuthology.orchestra.run.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:43.126 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:43.126 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:43.126 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2483304236 0 0) 0x7f28d813c720 con 0x7f28b80094f0 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 10 monclient(hunting): my global_id is 4106 2024-03-25T18:07:43.127 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.125+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f28b800cb60 con 0x7f28b80094f0 2024-03-25T18:07:43.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 939352318 0 0) 0x7f28d813c440 con 0x7f28b80060f0 2024-03-25T18:07:43.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 >> v1:172.21.15.37:6790/0 conn(0x7f28b80094f0 legacy=0x7f28d813ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:43.139 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 >> v1:172.21.15.117:6789/0 conn(0x7f28d80aaa10 legacy=0x7f28d81e1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28d4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28d4ff9640 1 monclient: found mon.a 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28d4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28d4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28d813ca00 con 0x7f28b80060f0 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28d4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28d4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28d4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28ddf54640 5 monclient: authenticate success, global_id 4107 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28ddf54640 10 monclient: _renew_subs 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28ddf54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28ddf54640 1 -- 172.21.15.37:0/884286135 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28d813b470 con 0x7f28b80060f0 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28ddf54640 10 client.4107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28ddf54640 10 monclient: _renew_subs 2024-03-25T18:07:43.140 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28ddf54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:43.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28ddf54640 1 -- 172.21.15.37:0/884286135 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f28d80ac370 con 0x7f28b80060f0 2024-03-25T18:07:43.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.138+0000 7f28ddf54640 1 librados: init done 2024-03-25T18:07:43.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.140+0000 7f28ddf54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:43.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.140+0000 7f28ddf54640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:43.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.140+0000 7f28ddf54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:43.141 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.140+0000 7f28ddf54640 1 -- 172.21.15.37:0/884286135 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28d80af960 con 0x7f28b80060f0 2024-03-25T18:07:43.376 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28c80049d0 con 0x7f28b80060f0 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f28c8004da0 con 0x7f28b80060f0 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 10 monclient: dump: 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:43.377 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:43.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f28c8005350 con 0x7f28b80060f0 2024-03-25T18:07:43.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d67fc640 4 set_mon_vals no callback set 2024-03-25T18:07:43.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3575901478 0 0) 0x7f28c8006460 con 0x7f28b80060f0 2024-03-25T18:07:43.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 10 client.4107.objecter ms_dispatch 0x7f28d8000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:07:43.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 3 client.4107.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:07:43.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 3 client.4107.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:07:43.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 20 client.4107.objecter dump_active .. 0 homeless 2024-03-25T18:07:43.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f28c8006ae0 con 0x7f28b80060f0 2024-03-25T18:07:43.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:43.378 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.375+0000 7f28d4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:43.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.445+0000 7f28ddf54640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:07:43.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.445+0000 7f28ddf54640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:07:43.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.445+0000 7f28ddf54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:43.446 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.445+0000 7f28ddf54640 1 -- 172.21.15.37:0/884286135 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f28d80af960 con 0x7f28b80060f0 2024-03-25T18:07:43.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.604+0000 7f28d4ff9640 1 -- 172.21.15.37:0/884286135 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f28c80056e0 con 0x7f28b80060f0 2024-03-25T18:07:43.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.604+0000 7f28d4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:07:43.605 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.604+0000 7f28d4ff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:07:43.605 INFO:teuthology.orchestra.run.smithi037.stderr:adjusted tunables profile to default 2024-03-25T18:07:43.606 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 10 librados: watch_flush enter 2024-03-25T18:07:43.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 10 librados: watch_flush exit 2024-03-25T18:07:43.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 20 client.4107.objecter shutdown clearing up homeless session... 2024-03-25T18:07:43.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 10 client.4107.objecter successfully canceled tick 2024-03-25T18:07:43.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 10 monclient: shutdown 2024-03-25T18:07:43.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:43.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 1 -- 172.21.15.37:0/884286135 >> v1:172.21.15.37:6789/0 conn(0x7f28b80060f0 legacy=0x7f28d8130860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:43.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 1 -- 172.21.15.37:0/884286135 shutdown_connections 2024-03-25T18:07:43.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 1 -- 172.21.15.37:0/884286135 >> 172.21.15.37:0/884286135 conn(0x7f28d806b560 msgr2=0x7f28d8071760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:43.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 1 -- 172.21.15.37:0/884286135 shutdown_connections 2024-03-25T18:07:43.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 1 -- 172.21.15.37:0/884286135 wait complete. 2024-03-25T18:07:43.607 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:43.605+0000 7f28ddf54640 1 librados: shutdown 2024-03-25T18:07:43.617 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:07:43.617 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:07:43.617 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:07:43.617 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:07:43.645 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:07:43.645 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:07:43.697 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:43.697+0000 7f4033642040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:07:43.712 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:07:43.712 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:07:43.751 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:43.751+0000 7f4033642040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:07:43.779 DEBUG:teuthology.orchestra.run.smithi037:> set -ex 2024-03-25T18:07:43.780 DEBUG:teuthology.orchestra.run.smithi037:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:07:43.847 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:07:43.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:07:43.873 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:07:43.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:07:43.929 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:43.929+0000 7f6850e56040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:07:43.940 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:07:43.940 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:07:43.992 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:43.992+0000 7f6850e56040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:07:44.008 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-03-25T18:07:44.008 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:07:44.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 9b8b387b-0c1c-40a3-9b30-bbde42ac181e 0 2024-03-25T18:07:44.215 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.215+0000 7fa2a0f72640 10 monclient: get_monmap_and_config 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.215+0000 7fa2a0f72640 10 monclient: build_initial_monmap 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.215+0000 7fa2a0f72640 10 monclient: monmap: 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:07:44.216241+0000 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:07:44.216241+0000 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:44.216 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.215+0000 7fa2a0f72640 1 Processor -- start 2024-03-25T18:07:44.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.215+0000 7fa2a0f72640 1 -- start start 2024-03-25T18:07:44.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.215+0000 7fa2a0f72640 10 monclient: init 2024-03-25T18:07:44.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.216+0000 7fa2a0f72640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:44.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.216+0000 7fa2a0f72640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:07:44.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.216+0000 7fa2a0f72640 10 monclient(hunting): picked mon.noname-b con 0x7fa29c0b4700 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:44.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.216+0000 7fa2a0f72640 10 monclient(hunting): picked mon.noname-a con 0x7fa29c142a20 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:44.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.216+0000 7fa2a0f72640 10 monclient(hunting): picked mon.noname-c con 0x7fa29c0a96b0 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:44.218 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.216+0000 7fa2a0f72640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29c0a9d60 con 0x7fa29c142a20 2024-03-25T18:07:44.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.216+0000 7fa2a0f72640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29c13ff10 con 0x7fa29c0a96b0 2024-03-25T18:07:44.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.216+0000 7fa2a0f72640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29c0aa1f0 con 0x7fa29c0b4700 2024-03-25T18:07:44.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.216+0000 7fa2a0f72640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:44.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.216+0000 7fa2a0f72640 10 monclient(hunting): authenticate will time out at 6819.346614s 2024-03-25T18:07:44.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.217+0000 7fa29ad9d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fa29c0a96b0 0x7fa29c13f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58834/0 (socket says 172.21.15.117:58834) 2024-03-25T18:07:44.219 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.217+0000 7fa29ad9d640 1 -- 172.21.15.117:0/3305934056 learned_addr learned my addr 172.21.15.117:0/3305934056 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:44.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 718411896 0 0) 0x7fa29c0aa1f0 con 0x7fa29c0b4700 2024-03-25T18:07:44.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.220 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 10 monclient(hunting): my global_id is 4115 2024-03-25T18:07:44.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa284003520 con 0x7fa29c0b4700 2024-03-25T18:07:44.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 774820382 0 0) 0x7fa29c13ff10 con 0x7fa29c0a96b0 2024-03-25T18:07:44.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:07:44.221 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.220+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29c0aa1f0 con 0x7fa29c0a96b0 2024-03-25T18:07:44.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.221+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2263823485 0 0) 0x7fa29c0a9d60 con 0x7fa29c142a20 2024-03-25T18:07:44.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.221+0000 7fa29959a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.221+0000 7fa29959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.221+0000 7fa29959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.221+0000 7fa29959a640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:07:44.222 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.221+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29c13ff10 con 0x7fa29c142a20 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 278768956 0 0) 0x7fa29c0aa1f0 con 0x7fa29c0a96b0 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 >> v1:172.21.15.37:6790/0 conn(0x7fa29c0b4700 legacy=0x7fa29c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 >> v1:172.21.15.37:6789/0 conn(0x7fa29c142a20 legacy=0x7fa29c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 10 monclient: _finish_hunting 0 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 1 monclient: found mon.noname-c 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c0b4ae0 con 0x7fa29c0a96b0 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 10 monclient: _finish_auth 0 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 10 monclient: _check_auth_tickets 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa2940019f0 con 0x7fa29c0a96b0 2024-03-25T18:07:44.225 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa2a0f72640 5 monclient: authenticate success, global_id 4114 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa2a0f72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa29959a640 10 monclient: dump: 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.226 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.224+0000 7fa2a0f72640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:44.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.249+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa294001b50 con 0x7fa29c0a96b0 2024-03-25T18:07:44.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.249+0000 7fa29959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:44.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.249+0000 7fa29959a640 1 -- 172.21.15.117:0/3305934056 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa294001f60 con 0x7fa29c0a96b0 2024-03-25T18:07:44.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.249+0000 7fa29959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.249+0000 7fa29959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:44.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.249+0000 7fa29959a640 10 monclient: dump: 2024-03-25T18:07:44.250 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:44.250 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:44.250 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.250 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.251 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.251 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:44.251 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:44.251 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:44.251 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:44.251 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.249+0000 7fa2a0f72640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:44.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.249+0000 7fa2a0f72640 4 set_mon_vals no callback set 2024-03-25T18:07:44.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.250+0000 7fa2a0f72640 10 monclient: shutdown 2024-03-25T18:07:44.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.250+0000 7fa2a0f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:44.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.250+0000 7fa2a0f72640 1 -- 172.21.15.117:0/3305934056 >> v1:172.21.15.117:6789/0 conn(0x7fa29c0a96b0 legacy=0x7fa29c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.250+0000 7fa2a0f72640 1 -- 172.21.15.117:0/3305934056 shutdown_connections 2024-03-25T18:07:44.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.250+0000 7fa2a0f72640 1 -- 172.21.15.117:0/3305934056 >> 172.21.15.117:0/3305934056 conn(0x7fa29c0668e0 msgr2=0x7fa29c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:44.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.250+0000 7fa2a0f72640 1 -- 172.21.15.117:0/3305934056 shutdown_connections 2024-03-25T18:07:44.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.250+0000 7fa2a0f72640 1 -- 172.21.15.117:0/3305934056 wait complete. 2024-03-25T18:07:44.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.251+0000 7fa2a0f72640 10 monclient: build_initial_monmap 2024-03-25T18:07:44.254 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.251+0000 7fa2a0f72640 10 monclient: monmap: 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.251+0000 7fa2a0f72640 1 librados: starting msgr at 2024-03-25T18:07:44.255 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.251+0000 7fa2a0f72640 1 librados: starting objecter 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 1 Processor -- start 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 1 -- start start 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 1 librados: setting wanted keys 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 1 librados: calling monclient init 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 10 monclient: init 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 10 monclient(hunting): picked mon.noname-b con 0x7fa29c0a96b0 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 10 monclient(hunting): picked mon.noname-a con 0x7fa29c0b4700 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 10 monclient(hunting): picked mon.noname-c con 0x7fa29c142a20 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29c13c3c0 con 0x7fa29c0b4700 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29c0ac100 con 0x7fa29c0a96b0 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa29c1d7280 con 0x7fa29c142a20 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.252+0000 7fa2a0f72640 10 monclient(hunting): authenticate will time out at 6819.382651s 2024-03-25T18:07:44.256 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.253+0000 7fa29a59c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fa29c0a96b0 0x7fa29c13bcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58836/0 (socket says 172.21.15.117:58836) 2024-03-25T18:07:44.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.253+0000 7fa29a59c640 1 -- 172.21.15.117:0/2636582666 learned_addr learned my addr 172.21.15.117:0/2636582666 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:44.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.253+0000 7fa28affd640 10 client.?.objecter ms_handle_connect 0x7fa29c0a96b0 2024-03-25T18:07:44.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.253+0000 7fa28affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:44.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.254+0000 7fa28affd640 10 client.?.objecter ms_handle_connect 0x7fa29c0b4700 2024-03-25T18:07:44.257 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.254+0000 7fa28affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.255+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3741959235 0 0) 0x7fa29c0ac100 con 0x7fa29c0a96b0 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.255+0000 7fa28affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.255+0000 7fa28affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.255+0000 7fa28affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.255+0000 7fa28affd640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.255+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa27c004eb0 con 0x7fa29c0a96b0 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.256+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3661087559 0 0) 0x7fa29c13c3c0 con 0x7fa29c0b4700 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.256+0000 7fa28affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.256+0000 7fa28affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.256+0000 7fa28affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.256+0000 7fa28affd640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:07:44.258 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.256+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa29c0ac100 con 0x7fa29c0b4700 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.261+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3840093073 0 0) 0x7fa29c0ac100 con 0x7fa29c0b4700 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.261+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa29c13c3c0 con 0x7fa29c0b4700 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.261+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa2900018f0 con 0x7fa29c0b4700 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.261+0000 7fa28affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.261+0000 7fa28affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.261+0000 7fa28affd640 10 monclient(hunting): dump: 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.262 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:44.263 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:44.263 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:44.263 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:44.263 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.261+0000 7fa28affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:44.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2551741762 0 0) 0x7fa29c13c3c0 con 0x7fa29c0b4700 2024-03-25T18:07:44.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 >> v1:172.21.15.37:6790/0 conn(0x7fa29c142a20 legacy=0x7fa29c1e1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 >> v1:172.21.15.117:6789/0 conn(0x7fa29c0a96b0 legacy=0x7fa29c13bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa28affd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa28affd640 1 monclient: found mon.a 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa28affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa28affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c1d8150 con 0x7fa29c0b4700 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa28affd640 10 monclient: _finish_auth 0 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa28affd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa28affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa2a0f72640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa2a0f72640 10 monclient: _renew_subs 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa2a0f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:44.267 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa2a0f72640 1 -- 172.21.15.117:0/2636582666 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa29c1d7520 con 0x7fa29c0b4700 2024-03-25T18:07:44.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa2a0f72640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:44.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa2a0f72640 10 monclient: _renew_subs 2024-03-25T18:07:44.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa2a0f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:44.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa2a0f72640 1 -- 172.21.15.117:0/2636582666 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa29c13b220 con 0x7fa29c0b4700 2024-03-25T18:07:44.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.265+0000 7fa2a0f72640 1 librados: init done 2024-03-25T18:07:44.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.266+0000 7fa2a0f72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:44.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.266+0000 7fa2a0f72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:44.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.266+0000 7fa2a0f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:44.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.266+0000 7fa2a0f72640 1 -- 172.21.15.117:0/2636582666 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa29c04f730 con 0x7fa29c0b4700 2024-03-25T18:07:44.307 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:44.306+0000 7f4033642040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:07:44.360 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:44.359+0000 7f4033642040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:07:44.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.369+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa290003610 con 0x7fa29c0b4700 2024-03-25T18:07:44.369 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.369+0000 7fa28affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.369+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa2900039a0 con 0x7fa29c0b4700 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.369+0000 7fa28affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.369+0000 7fa28affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.369+0000 7fa28affd640 10 monclient: dump: 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:44.370 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:44.371 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:44.371 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.369+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa290003f50 con 0x7fa29c0b4700 2024-03-25T18:07:44.371 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.369+0000 7fa298d99640 4 set_mon_vals no callback set 2024-03-25T18:07:44.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.370+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3028789936 0 0) 0x7fa2900052e0 con 0x7fa29c0b4700 2024-03-25T18:07:44.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.370+0000 7fa28affd640 10 client.4119.objecter ms_dispatch 0x7fa29c000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:07:44.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.370+0000 7fa28affd640 3 client.4119.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:07:44.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.370+0000 7fa28affd640 3 client.4119.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:07:44.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.370+0000 7fa28affd640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:07:44.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.370+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa290004640 con 0x7fa29c0b4700 2024-03-25T18:07:44.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.370+0000 7fa28affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:44.372 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.370+0000 7fa28affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:44.406 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:44.405+0000 7f4033642040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:07:44.443 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.443+0000 7fa2a0f72640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9b8b387b-0c1c-40a3-9b30-bbde42ac181e", "id": 0}] 2024-03-25T18:07:44.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.443+0000 7fa2a0f72640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9b8b387b-0c1c-40a3-9b30-bbde42ac181e", "id": 0}] 2024-03-25T18:07:44.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.443+0000 7fa2a0f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:44.444 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.443+0000 7fa2a0f72640 1 -- 172.21.15.117:0/2636582666 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9b8b387b-0c1c-40a3-9b30-bbde42ac181e", "id": 0} v 0) v1 -- 0x7fa29c069f30 con 0x7fa29c0b4700 2024-03-25T18:07:44.462 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:44.461+0000 7f4033642040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:07:44.504 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:44.503+0000 7f4033642040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:07:44.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.532+0000 7fa28affd640 1 -- 172.21.15.117:0/2636582666 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9b8b387b-0c1c-40a3-9b30-bbde42ac181e", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 1976504908 0 732707515) 0x7fa290004880 con 0x7fa29c0b4700 2024-03-25T18:07:44.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.532+0000 7fa28affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9b8b387b-0c1c-40a3-9b30-bbde42ac181e", "id": 0}] 2024-03-25T18:07:44.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.532+0000 7fa28affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:44.533 INFO:teuthology.orchestra.run.smithi117.stdout:0 2024-03-25T18:07:44.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 10 librados: watch_flush enter 2024-03-25T18:07:44.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 10 librados: watch_flush exit 2024-03-25T18:07:44.534 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:07:44.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:07:44.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 10 monclient: shutdown 2024-03-25T18:07:44.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:44.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 1 -- 172.21.15.117:0/2636582666 >> v1:172.21.15.37:6789/0 conn(0x7fa29c0b4700 legacy=0x7fa29c1d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 1 -- 172.21.15.117:0/2636582666 shutdown_connections 2024-03-25T18:07:44.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 1 -- 172.21.15.117:0/2636582666 >> 172.21.15.117:0/2636582666 conn(0x7fa29c0668e0 msgr2=0x7fa29c13d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:44.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 1 -- 172.21.15.117:0/2636582666 shutdown_connections 2024-03-25T18:07:44.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 1 -- 172.21.15.117:0/2636582666 wait complete. 2024-03-25T18:07:44.535 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.533+0000 7fa2a0f72640 1 librados: shutdown 2024-03-25T18:07:44.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new abc7c058-668d-4fef-90fa-3d0d17a0aa34 1 2024-03-25T18:07:44.556 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:44.556+0000 7f6850e56040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:07:44.613 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:44.613+0000 7f6850e56040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:07:44.627 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:44.626+0000 7f4033642040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:07:44.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.642+0000 7f75993ef640 10 monclient: get_monmap_and_config 2024-03-25T18:07:44.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.642+0000 7f75993ef640 10 monclient: build_initial_monmap 2024-03-25T18:07:44.643 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.642+0000 7f75993ef640 10 monclient: monmap: 2024-03-25T18:07:44.643 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:44.643 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:44.643 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:07:44.643798+0000 2024-03-25T18:07:44.643 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:07:44.643798+0000 2024-03-25T18:07:44.643 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:44.644 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:44.644 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:44.644 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:44.644 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:44.644 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.643+0000 7f75993ef640 1 Processor -- start 2024-03-25T18:07:44.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.643+0000 7f75993ef640 1 -- start start 2024-03-25T18:07:44.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.643+0000 7f75993ef640 10 monclient: init 2024-03-25T18:07:44.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75993ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:44.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75993ef640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:07:44.645 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75993ef640 10 monclient(hunting): picked mon.noname-b con 0x7f75940b1d40 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:44.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75993ef640 10 monclient(hunting): picked mon.noname-a con 0x7f75940b14e0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:44.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75993ef640 10 monclient(hunting): picked mon.noname-c con 0x7f75940b4400 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:44.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75993ef640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f759413ff90 con 0x7f75940b14e0 2024-03-25T18:07:44.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75993ef640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75940a9a60 con 0x7f75940b4400 2024-03-25T18:07:44.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75993ef640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75940ab4a0 con 0x7f75940b1d40 2024-03-25T18:07:44.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75993ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:44.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75993ef640 10 monclient(hunting): authenticate will time out at 6819.774061s 2024-03-25T18:07:44.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75937fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f75940b4400 0x7f759414dc60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58848/0 (socket says 172.21.15.117:58848) 2024-03-25T18:07:44.646 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.644+0000 7f75937fe640 1 -- 172.21.15.117:0/341274308 learned_addr learned my addr 172.21.15.117:0/341274308 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197542875 0 0) 0x7f75940a9a60 con 0x7f75940b4400 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7580003520 con 0x7f75940b4400 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3207626233 0 0) 0x7f75940ab4a0 con 0x7f75940b1d40 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 10 monclient(hunting): my global_id is 4118 2024-03-25T18:07:44.648 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.647+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75940a9a60 con 0x7f75940b1d40 2024-03-25T18:07:44.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.648+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3344413750 0 0) 0x7f759413ff90 con 0x7f75940b14e0 2024-03-25T18:07:44.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.648+0000 7f75927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.648+0000 7f75927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.648+0000 7f75927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.648+0000 7f75927fc640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:07:44.649 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.648+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75940ab4a0 con 0x7f75940b14e0 2024-03-25T18:07:44.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3381524471 0 0) 0x7f7580003520 con 0x7f75940b4400 2024-03-25T18:07:44.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 >> v1:172.21.15.37:6790/0 conn(0x7f75940b1d40 legacy=0x7f75940b0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 >> v1:172.21.15.37:6789/0 conn(0x7f75940b14e0 legacy=0x7f7594142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:44.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 1 monclient: found mon.noname-c 2024-03-25T18:07:44.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:44.651 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75940a9540 con 0x7f75940b4400 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f758c001a10 con 0x7f75940b4400 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75927fc640 10 monclient: dump: 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.652 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:44.653 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:44.653 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:44.653 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:44.653 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75993ef640 5 monclient: authenticate success, global_id 4120 2024-03-25T18:07:44.653 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.650+0000 7f75993ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.652+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f758c001b70 con 0x7f75940b4400 2024-03-25T18:07:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.652+0000 7f75927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.652+0000 7f75927fc640 1 -- 172.21.15.117:0/341274308 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f758c001f80 con 0x7f75940b4400 2024-03-25T18:07:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.652+0000 7f75993ef640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.652+0000 7f75993ef640 4 set_mon_vals no callback set 2024-03-25T18:07:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.652+0000 7f75927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.652+0000 7f75927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.652+0000 7f75927fc640 10 monclient: dump: 2024-03-25T18:07:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:44.655 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.653+0000 7f75993ef640 10 monclient: shutdown 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.653+0000 7f75993ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.653+0000 7f75993ef640 1 -- 172.21.15.117:0/341274308 >> v1:172.21.15.117:6789/0 conn(0x7f75940b4400 legacy=0x7f759414dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.653+0000 7f75993ef640 1 -- 172.21.15.117:0/341274308 shutdown_connections 2024-03-25T18:07:44.656 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.653+0000 7f75993ef640 1 -- 172.21.15.117:0/341274308 >> 172.21.15.117:0/341274308 conn(0x7f75940b7df0 msgr2=0x7f75940b81d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.653+0000 7f75993ef640 1 -- 172.21.15.117:0/341274308 shutdown_connections 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.653+0000 7f75993ef640 1 -- 172.21.15.117:0/341274308 wait complete. 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.654+0000 7f75993ef640 10 monclient: build_initial_monmap 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.654+0000 7f75993ef640 10 monclient: monmap: 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.654+0000 7f75993ef640 1 librados: starting msgr at 2024-03-25T18:07:44.657 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.654+0000 7f75993ef640 1 librados: starting objecter 2024-03-25T18:07:44.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.654+0000 7f75993ef640 1 Processor -- start 2024-03-25T18:07:44.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.654+0000 7f75993ef640 1 -- start start 2024-03-25T18:07:44.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.654+0000 7f75993ef640 1 librados: setting wanted keys 2024-03-25T18:07:44.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.654+0000 7f75993ef640 1 librados: calling monclient init 2024-03-25T18:07:44.658 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.654+0000 7f75993ef640 10 monclient: init 2024-03-25T18:07:44.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f75993ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:44.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f75993ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:44.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f75993ef640 10 monclient(hunting): picked mon.noname-b con 0x7f75940b14e0 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:44.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f75993ef640 10 monclient(hunting): picked mon.noname-a con 0x7f75940b1d40 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:44.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f75993ef640 10 monclient(hunting): picked mon.noname-c con 0x7f75940b4400 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:44.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f75993ef640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75940afee0 con 0x7f75940b1d40 2024-03-25T18:07:44.660 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f75993ef640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75941d7950 con 0x7f75940b14e0 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f75993ef640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75940acd60 con 0x7f75940b4400 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f75993ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f75993ef640 10 monclient(hunting): authenticate will time out at 6819.785115s 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f7592ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f75940b14e0 0x7f75941d7240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58862/0 (socket says 172.21.15.117:58862) 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f7592ffd640 1 -- 172.21.15.117:0/1353683105 learned_addr learned my addr 172.21.15.117:0/1353683105 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f758b7fe640 10 client.?.objecter ms_handle_connect 0x7f75940b14e0 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.655+0000 7f758b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.656+0000 7f758b7fe640 10 client.?.objecter ms_handle_connect 0x7f75940b4400 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.656+0000 7f758b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.656+0000 7f758b7fe640 10 client.?.objecter ms_handle_connect 0x7f75940b1d40 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.656+0000 7f758b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.657+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4281732827 0 0) 0x7f75941d7950 con 0x7f75940b14e0 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.657+0000 7f758b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.657+0000 7f758b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.661 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.657+0000 7f758b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.657+0000 7f758b7fe640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.657+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7574004eb0 con 0x7f75940b14e0 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4272736472 0 0) 0x7f75940acd60 con 0x7f75940b4400 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75941d7950 con 0x7f75940b4400 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2317857965 0 0) 0x7f75940afee0 con 0x7f75940b1d40 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:07:44.662 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.658+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f75940acd60 con 0x7f75940b1d40 2024-03-25T18:07:44.663 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:44.662+0000 7f6850e56040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:07:44.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.662+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1275262143 0 0) 0x7f7574004eb0 con 0x7f75940b14e0 2024-03-25T18:07:44.663 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.662+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f75940afee0 con 0x7f75940b14e0 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.662+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7584001a30 con 0x7f75940b14e0 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.662+0000 7f758b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.662+0000 7f758b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.662+0000 7f758b7fe640 10 monclient(hunting): dump: 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.664 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.662+0000 7f758b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:44.666 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.666+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3378630074 0 0) 0x7f75940acd60 con 0x7f75940b1d40 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.666+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7574004eb0 con 0x7f75940b1d40 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.666+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f757c001830 con 0x7f75940b1d40 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.666+0000 7f758b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.666+0000 7f758b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.666+0000 7f758b7fe640 10 monclient(hunting): dump: 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:44.667 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 763279487 0 0) 0x7f7574004eb0 con 0x7f75940b1d40 2024-03-25T18:07:44.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 >> v1:172.21.15.37:6790/0 conn(0x7f75940b4400 legacy=0x7f75940af7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.672 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 >> v1:172.21.15.117:6789/0 conn(0x7f75940b14e0 legacy=0x7f75941d7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f758b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f758b7fe640 1 monclient: found mon.a 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f758b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f758b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75940acef0 con 0x7f75940b1d40 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f758b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f758b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f758b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f75993ef640 5 monclient: authenticate success, global_id 4125 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f75993ef640 10 monclient: _renew_subs 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f75993ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f75993ef640 1 -- 172.21.15.117:0/1353683105 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75940ac7d0 con 0x7f75940b1d40 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f75993ef640 10 client.4125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f75993ef640 10 monclient: _renew_subs 2024-03-25T18:07:44.673 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f75993ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:44.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f75993ef640 1 -- 172.21.15.117:0/1353683105 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f75940ad980 con 0x7f75940b1d40 2024-03-25T18:07:44.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.671+0000 7f75993ef640 1 librados: init done 2024-03-25T18:07:44.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.672+0000 7f75993ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:44.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.672+0000 7f75993ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:44.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.672+0000 7f75993ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:44.674 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.672+0000 7f75993ef640 1 -- 172.21.15.117:0/1353683105 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75940ad4c0 con 0x7f75940b1d40 2024-03-25T18:07:44.700 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:44.700+0000 7f4033642040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:07:44.720 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:44.719+0000 7f6850e56040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:07:44.745 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:44.745+0000 7f4033642040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:07:44.764 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:44.763+0000 7f6850e56040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:07:44.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.778+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f757c001d40 con 0x7f75940b1d40 2024-03-25T18:07:44.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.778+0000 7f758b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:44.778 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.778+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f757c003610 con 0x7f75940b1d40 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.778+0000 7f758b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.778+0000 7f758b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.778+0000 7f758b7fe640 10 monclient: dump: 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:44.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.778+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f757c003bc0 con 0x7f75940b1d40 2024-03-25T18:07:44.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.778+0000 7f7591ffb640 4 set_mon_vals no callback set 2024-03-25T18:07:44.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.779+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 4245201835 0 0) 0x7f757c005130 con 0x7f75940b1d40 2024-03-25T18:07:44.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.779+0000 7f758b7fe640 10 client.4125.objecter ms_dispatch 0x7f7594000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:07:44.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.779+0000 7f758b7fe640 3 client.4125.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:07:44.780 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.779+0000 7f758b7fe640 3 client.4125.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:07:44.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.779+0000 7f758b7fe640 20 client.4125.objecter dump_active .. 0 homeless 2024-03-25T18:07:44.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.779+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f757c0040f0 con 0x7f75940b1d40 2024-03-25T18:07:44.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.779+0000 7f758b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:44.781 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.779+0000 7f758b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:44.798 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:44.797+0000 7f4033642040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:07:44.844 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:44.844+0000 7f4033642040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:07:44.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.850+0000 7f75993ef640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "abc7c058-668d-4fef-90fa-3d0d17a0aa34", "id": 1}] 2024-03-25T18:07:44.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.850+0000 7f75993ef640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "abc7c058-668d-4fef-90fa-3d0d17a0aa34", "id": 1}] 2024-03-25T18:07:44.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.850+0000 7f75993ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:44.851 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.850+0000 7f75993ef640 1 -- 172.21.15.117:0/1353683105 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd new", "uuid": "abc7c058-668d-4fef-90fa-3d0d17a0aa34", "id": 1} v 0) v1 -- 0x7f75940aea00 con 0x7f75940b1d40 2024-03-25T18:07:44.886 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:44.885+0000 7f6850e56040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:07:44.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.910+0000 7f758b7fe640 1 -- 172.21.15.117:0/1353683105 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "abc7c058-668d-4fef-90fa-3d0d17a0aa34", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 862189748 0 940483276) 0x7f757c004460 con 0x7f75940b1d40 2024-03-25T18:07:44.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.910+0000 7f758b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "abc7c058-668d-4fef-90fa-3d0d17a0aa34", "id": 1}] 2024-03-25T18:07:44.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.910+0000 7f758b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:44.911 INFO:teuthology.orchestra.run.smithi117.stdout:1 2024-03-25T18:07:44.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.910+0000 7f75993ef640 10 librados: watch_flush enter 2024-03-25T18:07:44.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.910+0000 7f75993ef640 10 librados: watch_flush exit 2024-03-25T18:07:44.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.910+0000 7f75993ef640 20 client.4125.objecter shutdown clearing up homeless session... 2024-03-25T18:07:44.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.910+0000 7f75993ef640 10 client.4125.objecter successfully canceled tick 2024-03-25T18:07:44.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.911+0000 7f75993ef640 10 monclient: shutdown 2024-03-25T18:07:44.912 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.911+0000 7f75993ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:44.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.911+0000 7f75993ef640 1 -- 172.21.15.117:0/1353683105 >> v1:172.21.15.37:6789/0 conn(0x7f75940b1d40 legacy=0x7f75940af0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:44.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.911+0000 7f75993ef640 1 -- 172.21.15.117:0/1353683105 shutdown_connections 2024-03-25T18:07:44.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.911+0000 7f75993ef640 1 -- 172.21.15.117:0/1353683105 >> 172.21.15.117:0/1353683105 conn(0x7f75940b7df0 msgr2=0x7f75940b6b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:44.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.911+0000 7f75993ef640 1 -- 172.21.15.117:0/1353683105 shutdown_connections 2024-03-25T18:07:44.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.911+0000 7f75993ef640 1 -- 172.21.15.117:0/1353683105 wait complete. 2024-03-25T18:07:44.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:44.911+0000 7f75993ef640 1 librados: shutdown 2024-03-25T18:07:44.922 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new ddddbd2b-0e5f-4930-911b-9bac20d19eeb 2 2024-03-25T18:07:44.959 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:44.959+0000 7f6850e56040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:07:45.007 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:45.007+0000 7f6850e56040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:07:45.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.021+0000 7fa4ac67b640 10 monclient: get_monmap_and_config 2024-03-25T18:07:45.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.021+0000 7fa4ac67b640 10 monclient: build_initial_monmap 2024-03-25T18:07:45.022 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.021+0000 7fa4ac67b640 10 monclient: monmap: 2024-03-25T18:07:45.022 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:45.022 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.022 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:07:45.022549+0000 2024-03-25T18:07:45.022 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:07:45.022549+0000 2024-03-25T18:07:45.022 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:45.022 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.023 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:45.023 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:45.023 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:45.023 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 1 Processor -- start 2024-03-25T18:07:45.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 1 -- start start 2024-03-25T18:07:45.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 10 monclient: init 2024-03-25T18:07:45.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:45.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:45.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a40b4700 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:45.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a4142a20 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:45.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a40a96b0 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:45.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40a9d60 con 0x7fa4a4142a20 2024-03-25T18:07:45.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a413ff10 con 0x7fa4a40b4700 2024-03-25T18:07:45.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a40aa1f0 con 0x7fa4a40a96b0 2024-03-25T18:07:45.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:45.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.022+0000 7fa4ac67b640 10 monclient(hunting): authenticate will time out at 6820.152770s 2024-03-25T18:07:45.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.023+0000 7fa4aa417640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fa4a40b4700 0x7fa4a4142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58864/0 (socket says 172.21.15.117:58864) 2024-03-25T18:07:45.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.023+0000 7fa4aa417640 1 -- 172.21.15.117:0/4036963050 learned_addr learned my addr 172.21.15.117:0/4036963050 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899916241 0 0) 0x7fa4a40aa1f0 con 0x7fa4a40a96b0 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa490003520 con 0x7fa4a40a96b0 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2112356846 0 0) 0x7fa4a413ff10 con 0x7fa4a40b4700 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.025+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a40aa1f0 con 0x7fa4a40b4700 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.026+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 56317499 0 0) 0x7fa4a40a9d60 con 0x7fa4a4142a20 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.026+0000 7fa4a9415640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.027 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.026+0000 7fa4a9415640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.026+0000 7fa4a9415640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.026+0000 7fa4a9415640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:07:45.028 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.026+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a413ff10 con 0x7fa4a4142a20 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1973042312 0 0) 0x7fa4a40aa1f0 con 0x7fa4a40b4700 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 >> v1:172.21.15.37:6790/0 conn(0x7fa4a40a96b0 legacy=0x7fa4a413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 >> v1:172.21.15.37:6789/0 conn(0x7fa4a4142a20 legacy=0x7fa4a413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 10 monclient: _finish_hunting 0 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 1 monclient: found mon.noname-c 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a40b4ae0 con 0x7fa4a40b4700 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 10 monclient: _finish_auth 0 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 10 monclient: _check_auth_tickets 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa4a0001670 con 0x7fa4a40b4700 2024-03-25T18:07:45.030 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4a9415640 10 monclient: dump: 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4ac67b640 5 monclient: authenticate success, global_id 4126 2024-03-25T18:07:45.031 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.029+0000 7fa4ac67b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:45.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.030+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa4a00017d0 con 0x7fa4a40b4700 2024-03-25T18:07:45.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.030+0000 7fa4a9415640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:45.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.030+0000 7fa4a9415640 1 -- 172.21.15.117:0/4036963050 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa4a0001be0 con 0x7fa4a40b4700 2024-03-25T18:07:45.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.030+0000 7fa4a9415640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.032 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.030+0000 7fa4a9415640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.030+0000 7fa4a9415640 10 monclient: dump: 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.030+0000 7fa4ac67b640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.030+0000 7fa4ac67b640 4 set_mon_vals no callback set 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.031+0000 7fa4ac67b640 10 monclient: shutdown 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.031+0000 7fa4ac67b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:45.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.031+0000 7fa4ac67b640 1 -- 172.21.15.117:0/4036963050 >> v1:172.21.15.117:6789/0 conn(0x7fa4a40b4700 legacy=0x7fa4a4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.031+0000 7fa4ac67b640 1 -- 172.21.15.117:0/4036963050 shutdown_connections 2024-03-25T18:07:45.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.032+0000 7fa4ac67b640 1 -- 172.21.15.117:0/4036963050 >> 172.21.15.117:0/4036963050 conn(0x7fa4a40668e0 msgr2=0x7fa4a4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:45.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.032+0000 7fa4ac67b640 1 -- 172.21.15.117:0/4036963050 shutdown_connections 2024-03-25T18:07:45.034 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.032+0000 7fa4ac67b640 1 -- 172.21.15.117:0/4036963050 wait complete. 2024-03-25T18:07:45.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.032+0000 7fa4ac67b640 10 monclient: build_initial_monmap 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.032+0000 7fa4ac67b640 10 monclient: monmap: 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.033+0000 7fa4ac67b640 1 librados: starting msgr at 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.033+0000 7fa4ac67b640 1 librados: starting objecter 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.033+0000 7fa4ac67b640 1 Processor -- start 2024-03-25T18:07:45.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.033+0000 7fa4ac67b640 1 -- start start 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.033+0000 7fa4ac67b640 1 librados: setting wanted keys 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.033+0000 7fa4ac67b640 1 librados: calling monclient init 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.033+0000 7fa4ac67b640 10 monclient: init 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.034+0000 7fa4ac67b640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.034+0000 7fa4ac67b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.034+0000 7fa4ac67b640 10 monclient(hunting): picked mon.noname-a con 0x7fa4a40a96b0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.034+0000 7fa4ac67b640 10 monclient(hunting): picked mon.noname-c con 0x7fa4a40b4700 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.034+0000 7fa4ac67b640 10 monclient(hunting): picked mon.noname-b con 0x7fa4a4142a20 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.034+0000 7fa4ac67b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a4135f10 con 0x7fa4a40a96b0 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.034+0000 7fa4ac67b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a4139d70 con 0x7fa4a4142a20 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.034+0000 7fa4ac67b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4a413a050 con 0x7fa4a40b4700 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.034+0000 7fa4ac67b640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:45.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.034+0000 7fa4ac67b640 10 monclient(hunting): authenticate will time out at 6820.164476s 2024-03-25T18:07:45.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.035+0000 7fa4aac18640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fa4a4142a20 0x7fa4a41dba30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58880/0 (socket says 172.21.15.117:58880) 2024-03-25T18:07:45.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.035+0000 7fa4aac18640 1 -- 172.21.15.117:0/341941605 learned_addr learned my addr 172.21.15.117:0/341941605 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:45.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.035+0000 7fa49affd640 10 client.?.objecter ms_handle_connect 0x7fa4a4142a20 2024-03-25T18:07:45.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.035+0000 7fa49affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:45.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.035+0000 7fa49affd640 10 client.?.objecter ms_handle_connect 0x7fa4a40b4700 2024-03-25T18:07:45.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.035+0000 7fa49affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.036+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159694142 0 0) 0x7fa4a4139d70 con 0x7fa4a4142a20 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.036+0000 7fa49affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.036+0000 7fa49affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.036+0000 7fa49affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.036+0000 7fa49affd640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.036+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa488004eb0 con 0x7fa4a4142a20 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.037+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1137229958 0 0) 0x7fa4a413a050 con 0x7fa4a40b4700 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.037+0000 7fa49affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.037+0000 7fa49affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.037+0000 7fa49affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.037+0000 7fa49affd640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:07:45.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.037+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa4a4139d70 con 0x7fa4a40b4700 2024-03-25T18:07:45.042 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.041+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1078667975 0 0) 0x7fa488004eb0 con 0x7fa4a4142a20 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.041+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa4a413a050 con 0x7fa4a4142a20 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.041+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa49c001c30 con 0x7fa4a4142a20 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.041+0000 7fa49affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.041+0000 7fa49affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.041+0000 7fa49affd640 10 monclient(hunting): dump: 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.043 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.042+0000 7fa49affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.042+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1577799718 0 0) 0x7fa4a4139d70 con 0x7fa4a40b4700 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.042+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa488004eb0 con 0x7fa4a40b4700 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.042+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa4940018b0 con 0x7fa4a40b4700 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.042+0000 7fa49affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.042+0000 7fa49affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.042+0000 7fa49affd640 10 monclient(hunting): dump: 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.044 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.045 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.045 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.045 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 522001219 0 0) 0x7fa4a413a050 con 0x7fa4a4142a20 2024-03-25T18:07:45.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 >> v1:172.21.15.37:6790/0 conn(0x7fa4a40b4700 legacy=0x7fa4a41db320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 >> v1:172.21.15.37:6789/0 conn(0x7fa4a40a96b0 legacy=0x7fa4a4130020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa49affd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:45.046 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa49affd640 1 monclient: found mon.b 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa49affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa49affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4a41363f0 con 0x7fa4a4142a20 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa49affd640 10 monclient: _finish_auth 0 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa49affd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa49affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa4ac67b640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa4ac67b640 10 monclient: _renew_subs 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa4ac67b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa4ac67b640 1 -- 172.21.15.117:0/341941605 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4a4138d80 con 0x7fa4a4142a20 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa4ac67b640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa4ac67b640 10 monclient: _renew_subs 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa4ac67b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa4ac67b640 1 -- 172.21.15.117:0/341941605 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4a4138670 con 0x7fa4a4142a20 2024-03-25T18:07:45.047 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.045+0000 7fa4ac67b640 1 librados: init done 2024-03-25T18:07:45.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.046+0000 7fa4ac67b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:45.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.046+0000 7fa4ac67b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:45.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.046+0000 7fa4ac67b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:45.048 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.046+0000 7fa4ac67b640 1 -- 172.21.15.117:0/341941605 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4a41360e0 con 0x7fa4a4142a20 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.066+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa49c0020e0 con 0x7fa4a4142a20 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.066+0000 7fa49affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.066+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa49c003b70 con 0x7fa4a4142a20 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.066+0000 7fa49affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.066+0000 7fa49affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.066+0000 7fa49affd640 10 monclient: dump: 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.067 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.068 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.068 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.068 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.066+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fa49c004120 con 0x7fa4a4142a20 2024-03-25T18:07:45.068 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.066+0000 7fa4a8c14640 4 set_mon_vals no callback set 2024-03-25T18:07:45.068 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:45.067+0000 7f6850e56040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:07:45.116 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:45.116+0000 7f6850e56040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:07:45.246 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:45.245+0000 7f4033642040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:07:45.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.249+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2755931078 0 0) 0x7fa49c005b40 con 0x7fa4a4142a20 2024-03-25T18:07:45.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.249+0000 7fa49affd640 10 client.4129.objecter ms_dispatch 0x7fa4a4000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:07:45.250 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.249+0000 7fa49affd640 3 client.4129.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:07:45.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.249+0000 7fa49affd640 3 client.4129.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:07:45.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.249+0000 7fa49affd640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T18:07:45.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.249+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fa49c0047f0 con 0x7fa4a4142a20 2024-03-25T18:07:45.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.249+0000 7fa49affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:45.251 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.249+0000 7fa49affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:45.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.344+0000 7fa4ac67b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ddddbd2b-0e5f-4930-911b-9bac20d19eeb", "id": 2}] 2024-03-25T18:07:45.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.344+0000 7fa4ac67b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ddddbd2b-0e5f-4930-911b-9bac20d19eeb", "id": 2}] 2024-03-25T18:07:45.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.344+0000 7fa4ac67b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:45.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.344+0000 7fa4ac67b640 1 -- 172.21.15.117:0/341941605 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ddddbd2b-0e5f-4930-911b-9bac20d19eeb", "id": 2} v 0) v1 -- 0x7fa4a4141220 con 0x7fa4a4142a20 2024-03-25T18:07:45.401 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:45.400+0000 7f4033642040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:07:45.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.447+0000 7fa49affd640 1 -- 172.21.15.117:0/341941605 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ddddbd2b-0e5f-4930-911b-9bac20d19eeb", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 1673019922 0 216599125) 0x7fa49c006060 con 0x7fa4a4142a20 2024-03-25T18:07:45.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.447+0000 7fa49affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ddddbd2b-0e5f-4930-911b-9bac20d19eeb", "id": 2}] 2024-03-25T18:07:45.448 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.447+0000 7fa49affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:45.448 INFO:teuthology.orchestra.run.smithi117.stdout:2 2024-03-25T18:07:45.449 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 10 librados: watch_flush enter 2024-03-25T18:07:45.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 10 librados: watch_flush exit 2024-03-25T18:07:45.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T18:07:45.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 10 client.4129.objecter successfully canceled tick 2024-03-25T18:07:45.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 10 monclient: shutdown 2024-03-25T18:07:45.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:45.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 1 -- 172.21.15.117:0/341941605 >> v1:172.21.15.117:6789/0 conn(0x7fa4a4142a20 legacy=0x7fa4a41dba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 1 -- 172.21.15.117:0/341941605 shutdown_connections 2024-03-25T18:07:45.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 1 -- 172.21.15.117:0/341941605 >> 172.21.15.117:0/341941605 conn(0x7fa4a40668e0 msgr2=0x7fa4a4130c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:45.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 1 -- 172.21.15.117:0/341941605 shutdown_connections 2024-03-25T18:07:45.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 1 -- 172.21.15.117:0/341941605 wait complete. 2024-03-25T18:07:45.450 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.448+0000 7fa4ac67b640 1 librados: shutdown 2024-03-25T18:07:45.459 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new e00e8f64-0c01-4f09-a358-ecc0a60e53f4 3 2024-03-25T18:07:45.515 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:45.514+0000 7f4033642040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:07:45.549 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:45.548+0000 7f6850e56040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:07:45.558 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.558+0000 7fbc50dc6640 10 monclient: get_monmap_and_config 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.558+0000 7fbc50dc6640 10 monclient: build_initial_monmap 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.558+0000 7fbc50dc6640 10 monclient: monmap: 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:07:45.559148+0000 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:07:45.559148+0000 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:45.559 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.558+0000 7fbc50dc6640 1 Processor -- start 2024-03-25T18:07:45.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.558+0000 7fbc50dc6640 1 -- start start 2024-03-25T18:07:45.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.558+0000 7fbc50dc6640 10 monclient: init 2024-03-25T18:07:45.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.559+0000 7fbc50dc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:45.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.559+0000 7fbc50dc6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:45.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.559+0000 7fbc50dc6640 10 monclient(hunting): picked mon.noname-a con 0x7fbc4c0b4700 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:45.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.559+0000 7fbc50dc6640 10 monclient(hunting): picked mon.noname-c con 0x7fbc4c142a20 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:45.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.559+0000 7fbc50dc6640 10 monclient(hunting): picked mon.noname-b con 0x7fbc4c0a96b0 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:45.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.559+0000 7fbc50dc6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c0a9d60 con 0x7fbc4c0b4700 2024-03-25T18:07:45.561 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.559+0000 7fbc50dc6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c13ff10 con 0x7fbc4c142a20 2024-03-25T18:07:45.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.559+0000 7fbc50dc6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c0aa1f0 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.559+0000 7fbc50dc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:45.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.559+0000 7fbc50dc6640 10 monclient(hunting): authenticate will time out at 6820.689482s 2024-03-25T18:07:45.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.560+0000 7fbc49d9b640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fbc4c142a20 0x7fbc4c13f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58886/0 (socket says 172.21.15.117:58886) 2024-03-25T18:07:45.562 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.560+0000 7fbc49d9b640 1 -- 172.21.15.117:0/1136416592 learned_addr learned my addr 172.21.15.117:0/1136416592 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:45.563 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:45.562+0000 7f4033642040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:07:45.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164277644 0 0) 0x7fbc4c0a9d60 con 0x7fbc4c0b4700 2024-03-25T18:07:45.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:07:45.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc34003520 con 0x7fbc4c0b4700 2024-03-25T18:07:45.563 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2534701420 0 0) 0x7fbc4c13ff10 con 0x7fbc4c142a20 2024-03-25T18:07:45.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:07:45.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.562+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4c0a9d60 con 0x7fbc4c142a20 2024-03-25T18:07:45.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.563+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1777980163 0 0) 0x7fbc4c0aa1f0 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.563+0000 7fbc4959a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.563+0000 7fbc4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.563+0000 7fbc4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.563+0000 7fbc4959a640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:07:45.564 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.563+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4c13ff10 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2211554485 0 0) 0x7fbc34003520 con 0x7fbc4c0b4700 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 >> v1:172.21.15.37:6790/0 conn(0x7fbc4c0a96b0 legacy=0x7fbc4c13f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 >> v1:172.21.15.117:6789/0 conn(0x7fbc4c142a20 legacy=0x7fbc4c13f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 10 monclient: _finish_hunting 0 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 1 monclient: found mon.noname-a 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c0b4ae0 con 0x7fbc4c0b4700 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 10 monclient: _finish_auth 0 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 10 monclient: _check_auth_tickets 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:45.567 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbc40001690 con 0x7fbc4c0b4700 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc50dc6640 5 monclient: authenticate success, global_id 4131 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.565+0000 7fbc4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.566+0000 7fbc4959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.566+0000 7fbc4959a640 10 monclient: dump: 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.568 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.569 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.566+0000 7fbc50dc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:45.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.567+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc400017f0 con 0x7fbc4c0b4700 2024-03-25T18:07:45.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.567+0000 7fbc4959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:45.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.567+0000 7fbc4959a640 1 -- 172.21.15.117:0/1136416592 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbc40001c00 con 0x7fbc4c0b4700 2024-03-25T18:07:45.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.567+0000 7fbc4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.567+0000 7fbc4959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:45.571 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.567+0000 7fbc4959a640 10 monclient: dump: 2024-03-25T18:07:45.571 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.567+0000 7fbc50dc6640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.567+0000 7fbc50dc6640 4 set_mon_vals no callback set 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.568+0000 7fbc50dc6640 10 monclient: shutdown 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.568+0000 7fbc50dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:45.572 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.568+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1136416592 >> v1:172.21.15.37:6789/0 conn(0x7fbc4c0b4700 legacy=0x7fbc4c142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.568+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1136416592 shutdown_connections 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.568+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1136416592 >> 172.21.15.117:0/1136416592 conn(0x7fbc4c0668e0 msgr2=0x7fbc4c066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.568+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1136416592 shutdown_connections 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.568+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1136416592 wait complete. 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.569+0000 7fbc50dc6640 10 monclient: build_initial_monmap 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.569+0000 7fbc50dc6640 10 monclient: monmap: 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.573 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.569+0000 7fbc50dc6640 1 librados: starting msgr at 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.569+0000 7fbc50dc6640 1 librados: starting objecter 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.569+0000 7fbc50dc6640 1 Processor -- start 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.569+0000 7fbc50dc6640 1 -- start start 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.569+0000 7fbc50dc6640 1 librados: setting wanted keys 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.569+0000 7fbc50dc6640 1 librados: calling monclient init 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.569+0000 7fbc50dc6640 10 monclient: init 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc50dc6640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc50dc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc50dc6640 10 monclient(hunting): picked mon.noname-a con 0x7fbc4c0a96b0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc50dc6640 10 monclient(hunting): picked mon.noname-c con 0x7fbc4c0b4700 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc50dc6640 10 monclient(hunting): picked mon.noname-b con 0x7fbc4c142a20 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc50dc6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c0aba20 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc50dc6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c0ad830 con 0x7fbc4c142a20 2024-03-25T18:07:45.574 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc50dc6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc4c1db6a0 con 0x7fbc4c0b4700 2024-03-25T18:07:45.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc50dc6640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:45.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc50dc6640 10 monclient(hunting): authenticate will time out at 6820.700097s 2024-03-25T18:07:45.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc4a59c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fbc4c0a96b0 0x7fbc4c1d4780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.117:56514/0 (socket says 172.21.15.117:56514) 2024-03-25T18:07:45.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc4a59c640 1 -- 172.21.15.117:0/1363074404 learned_addr learned my addr 172.21.15.117:0/1363074404 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:45.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.570+0000 7fbc3affd640 10 client.?.objecter ms_handle_connect 0x7fbc4c0a96b0 2024-03-25T18:07:45.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.571+0000 7fbc3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:45.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.571+0000 7fbc3affd640 10 client.?.objecter ms_handle_connect 0x7fbc4c0b4700 2024-03-25T18:07:45.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.571+0000 7fbc3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:45.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.571+0000 7fbc3affd640 10 client.?.objecter ms_handle_connect 0x7fbc4c142a20 2024-03-25T18:07:45.575 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.571+0000 7fbc3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:45.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5365067 0 0) 0x7fbc4c0aba20 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.576 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc2c004eb0 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2075558857 0 0) 0x7fbc4c0ad830 con 0x7fbc4c142a20 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4c0aba20 con 0x7fbc4c142a20 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3634107150 0 0) 0x7fbc4c1db6a0 con 0x7fbc4c0b4700 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:07:45.577 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.573+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc4c0ad830 con 0x7fbc4c0b4700 2024-03-25T18:07:45.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.577+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 182308843 0 0) 0x7fbc2c004eb0 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.577+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc4c1db6a0 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.577+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbc40001bc0 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.577+0000 7fbc3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.577+0000 7fbc3affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:45.578 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.577+0000 7fbc3affd640 10 monclient(hunting): dump: 2024-03-25T18:07:45.578 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.579 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.579 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.579 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.579 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.579 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.579 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.579 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.579 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.579 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.579 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.577+0000 7fbc3affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.578+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 382012759 0 0) 0x7fbc4c0aba20 con 0x7fbc4c142a20 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.578+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc2c004eb0 con 0x7fbc4c142a20 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.578+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbc44001b50 con 0x7fbc4c142a20 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.578+0000 7fbc3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.578+0000 7fbc3affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.578+0000 7fbc3affd640 10 monclient(hunting): dump: 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.580 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.581 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.581 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.581 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.581 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.580+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2040998532 0 0) 0x7fbc4c0ad830 con 0x7fbc4c0b4700 2024-03-25T18:07:45.582 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.580+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbc4c0aba20 con 0x7fbc4c0b4700 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.580+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbc3c0018f0 con 0x7fbc4c0b4700 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.580+0000 7fbc3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.580+0000 7fbc3affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.580+0000 7fbc3affd640 10 monclient(hunting): dump: 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1946456495 0 0) 0x7fbc4c1db6a0 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.583 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 >> v1:172.21.15.37:6790/0 conn(0x7fbc4c0b4700 legacy=0x7fbc4c1da880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 >> v1:172.21.15.117:6789/0 conn(0x7fbc4c142a20 legacy=0x7fbc4c1daf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc3affd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc3affd640 1 monclient: found mon.a 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc3affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc4c1d4e90 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc3affd640 10 monclient: _finish_auth 0 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc3affd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc50dc6640 5 monclient: authenticate success, global_id 4134 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc50dc6640 10 monclient: _renew_subs 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc50dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1363074404 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc4c13ba80 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc50dc6640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc50dc6640 10 monclient: _renew_subs 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc50dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:45.584 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1363074404 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc4c0ac790 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.581+0000 7fbc50dc6640 1 librados: init done 2024-03-25T18:07:45.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.582+0000 7fbc50dc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:45.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.582+0000 7fbc50dc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:45.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.582+0000 7fbc50dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:45.585 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.582+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1363074404 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc4c0ab880 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.678 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:45.677+0000 7f4033642040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:07:45.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.685+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbc400011f0 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.685+0000 7fbc3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:45.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.685+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbc40003960 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.686 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.685+0000 7fbc3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.686+0000 7fbc3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.686+0000 7fbc48d99640 4 set_mon_vals no callback set 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.686+0000 7fbc3affd640 10 monclient: dump: 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.687 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.686+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fbc40003f10 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.686+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2757427744 0 0) 0x7fbc40006610 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.686+0000 7fbc3affd640 10 client.4134.objecter ms_dispatch 0x7fbc4c000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:07:45.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.686+0000 7fbc3affd640 3 client.4134.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:07:45.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.686+0000 7fbc3affd640 3 client.4134.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:07:45.688 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.687+0000 7fbc3affd640 20 client.4134.objecter dump_active .. 0 homeless 2024-03-25T18:07:45.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.687+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fbc400043f0 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.687+0000 7fbc3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:45.689 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.687+0000 7fbc3affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:45.698 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:45.698+0000 7f6850e56040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:07:45.759 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.759+0000 7fbc50dc6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "e00e8f64-0c01-4f09-a358-ecc0a60e53f4", "id": 3}] 2024-03-25T18:07:45.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.759+0000 7fbc50dc6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "e00e8f64-0c01-4f09-a358-ecc0a60e53f4", "id": 3}] 2024-03-25T18:07:45.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.759+0000 7fbc50dc6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:45.760 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.759+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1363074404 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd new", "uuid": "e00e8f64-0c01-4f09-a358-ecc0a60e53f4", "id": 3} v 0) v1 -- 0x7fbc4c04f730 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.774 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:45.774+0000 7f4033642040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:07:45.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.794+0000 7fbc3affd640 1 -- 172.21.15.117:0/1363074404 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "e00e8f64-0c01-4f09-a358-ecc0a60e53f4", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 1468825750 0 525046306) 0x7fbc40006330 con 0x7fbc4c0a96b0 2024-03-25T18:07:45.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.794+0000 7fbc3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "e00e8f64-0c01-4f09-a358-ecc0a60e53f4", "id": 3}] 2024-03-25T18:07:45.795 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.794+0000 7fbc3affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:45.795 INFO:teuthology.orchestra.run.smithi117.stdout:3 2024-03-25T18:07:45.796 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.795+0000 7fbc50dc6640 10 librados: watch_flush enter 2024-03-25T18:07:45.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.795+0000 7fbc50dc6640 10 librados: watch_flush exit 2024-03-25T18:07:45.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.795+0000 7fbc50dc6640 20 client.4134.objecter shutdown clearing up homeless session... 2024-03-25T18:07:45.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.795+0000 7fbc50dc6640 10 client.4134.objecter successfully canceled tick 2024-03-25T18:07:45.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.795+0000 7fbc50dc6640 10 monclient: shutdown 2024-03-25T18:07:45.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.795+0000 7fbc50dc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:45.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.795+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1363074404 >> v1:172.21.15.37:6789/0 conn(0x7fbc4c0a96b0 legacy=0x7fbc4c1d4780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.796+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1363074404 shutdown_connections 2024-03-25T18:07:45.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.796+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1363074404 >> 172.21.15.117:0/1363074404 conn(0x7fbc4c0668e0 msgr2=0x7fbc4c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:45.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.796+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1363074404 shutdown_connections 2024-03-25T18:07:45.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.796+0000 7fbc50dc6640 1 -- 172.21.15.117:0/1363074404 wait complete. 2024-03-25T18:07:45.797 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.796+0000 7fbc50dc6640 1 librados: shutdown 2024-03-25T18:07:45.810 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d 4 2024-03-25T18:07:45.830 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:45.829+0000 7f6850e56040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:07:45.879 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:45.878+0000 7f6850e56040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:07:45.910 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.910+0000 7fee55754640 10 monclient: get_monmap_and_config 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.910+0000 7fee55754640 10 monclient: build_initial_monmap 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.910+0000 7fee55754640 10 monclient: monmap: 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:07:45.911281+0000 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:07:45.911281+0000 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:45.911 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 1 Processor -- start 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 1 -- start start 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 10 monclient: init 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 10 monclient(hunting): picked mon.noname-b con 0x7fee500b4700 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 10 monclient(hunting): picked mon.noname-c con 0x7fee50142a20 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 10 monclient(hunting): picked mon.noname-a con 0x7fee500a96b0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee500a9d60 con 0x7fee500a96b0 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5013ff10 con 0x7fee50142a20 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee500aa1f0 con 0x7fee500b4700 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.911+0000 7fee55754640 10 monclient(hunting): authenticate will time out at 6821.041616s 2024-03-25T18:07:45.913 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.912+0000 7fee4effd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fee500b4700 0x7fee50142310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.117:49008/0 (socket says 172.21.15.117:49008) 2024-03-25T18:07:45.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.912+0000 7fee4f7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fee500a96b0 0x7fee5013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.117:56530/0 (socket says 172.21.15.117:56530) 2024-03-25T18:07:45.914 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.912+0000 7fee4effd640 1 -- 172.21.15.117:0/240872988 learned_addr learned my addr 172.21.15.117:0/240872988 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:45.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2789240994 0 0) 0x7fee5013ff10 con 0x7fee50142a20 2024-03-25T18:07:45.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:07:45.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee3c003520 con 0x7fee50142a20 2024-03-25T18:07:45.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968795961 0 0) 0x7fee500aa1f0 con 0x7fee500b4700 2024-03-25T18:07:45.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.916 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:07:45.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee5013ff10 con 0x7fee500b4700 2024-03-25T18:07:45.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5930691 0 0) 0x7fee500a9d60 con 0x7fee500a96b0 2024-03-25T18:07:45.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:07:45.917 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.915+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee500aa1f0 con 0x7fee500a96b0 2024-03-25T18:07:45.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 759488570 0 0) 0x7fee3c003520 con 0x7fee50142a20 2024-03-25T18:07:45.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 >> v1:172.21.15.37:6790/0 conn(0x7fee500b4700 legacy=0x7fee50142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 >> v1:172.21.15.37:6789/0 conn(0x7fee500a96b0 legacy=0x7fee5013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:45.919 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee500b4ae0 con 0x7fee50142a20 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fee38001690 con 0x7fee50142a20 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee55754640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee55754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee4e7fc640 10 monclient: dump: 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.920 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.921 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.921 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.921 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.921 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.921 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.921 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.921 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.918+0000 7fee55754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:45.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.920+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee380017f0 con 0x7fee50142a20 2024-03-25T18:07:45.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.920+0000 7fee4e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:45.923 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.920+0000 7fee4e7fc640 1 -- 172.21.15.117:0/240872988 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fee38001c40 con 0x7fee50142a20 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.920+0000 7fee55754640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.920+0000 7fee55754640 4 set_mon_vals no callback set 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.920+0000 7fee4e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.920+0000 7fee4e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.920+0000 7fee4e7fc640 10 monclient: dump: 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.924 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.921+0000 7fee55754640 10 monclient: shutdown 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.921+0000 7fee55754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.921+0000 7fee55754640 1 -- 172.21.15.117:0/240872988 >> v1:172.21.15.117:6789/0 conn(0x7fee50142a20 legacy=0x7fee5013f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.921+0000 7fee55754640 1 -- 172.21.15.117:0/240872988 shutdown_connections 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.921+0000 7fee55754640 1 -- 172.21.15.117:0/240872988 >> 172.21.15.117:0/240872988 conn(0x7fee500668e0 msgr2=0x7fee50066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.921+0000 7fee55754640 1 -- 172.21.15.117:0/240872988 shutdown_connections 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.921+0000 7fee55754640 1 -- 172.21.15.117:0/240872988 wait complete. 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.921+0000 7fee55754640 10 monclient: build_initial_monmap 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.921+0000 7fee55754640 10 monclient: monmap: 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-25T18:07:45.925 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 1 librados: starting msgr at 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 1 librados: starting objecter 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 1 Processor -- start 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 1 -- start start 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 1 librados: setting wanted keys 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 1 librados: calling monclient init 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 10 monclient: init 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 10 monclient(hunting): picked mon.noname-c con 0x7fee500a96b0 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:45.926 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 10 monclient(hunting): picked mon.noname-b con 0x7fee500b4700 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:45.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 10 monclient(hunting): picked mon.noname-a con 0x7fee50142a20 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:45.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee501d7320 con 0x7fee50142a20 2024-03-25T18:07:45.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5012b3a0 con 0x7fee500b4700 2024-03-25T18:07:45.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee5012b530 con 0x7fee500a96b0 2024-03-25T18:07:45.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:45.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.922+0000 7fee55754640 10 monclient(hunting): authenticate will time out at 6821.052699s 2024-03-25T18:07:45.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.923+0000 7fee4f7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fee50142a20 0x7fee501e1cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.117:56540/0 (socket says 172.21.15.117:56540) 2024-03-25T18:07:45.927 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.923+0000 7fee4f7fe640 1 -- 172.21.15.117:0/1896123238 learned_addr learned my addr 172.21.15.117:0/1896123238 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:45.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.923+0000 7fee477fe640 10 client.?.objecter ms_handle_connect 0x7fee50142a20 2024-03-25T18:07:45.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.923+0000 7fee477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:45.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.923+0000 7fee477fe640 10 client.?.objecter ms_handle_connect 0x7fee500a96b0 2024-03-25T18:07:45.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.923+0000 7fee477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:45.928 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.923+0000 7fee477fe640 10 client.?.objecter ms_handle_connect 0x7fee500b4700 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.923+0000 7fee477fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136841420 0 0) 0x7fee5012b530 con 0x7fee500a96b0 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee30004eb0 con 0x7fee500a96b0 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454551954 0 0) 0x7fee5012b3a0 con 0x7fee500b4700 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.925+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee5012b530 con 0x7fee500b4700 2024-03-25T18:07:45.929 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.926+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3409144957 0 0) 0x7fee501d7320 con 0x7fee50142a20 2024-03-25T18:07:45.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.926+0000 7fee477fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:45.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.926+0000 7fee477fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:45.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.926+0000 7fee477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:45.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.926+0000 7fee477fe640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:07:45.930 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.926+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee5012b3a0 con 0x7fee50142a20 2024-03-25T18:07:45.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3595856788 0 0) 0x7fee30004eb0 con 0x7fee500a96b0 2024-03-25T18:07:45.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee501d7320 con 0x7fee500a96b0 2024-03-25T18:07:45.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fee40001a50 con 0x7fee500a96b0 2024-03-25T18:07:45.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:45.932 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 10 monclient(hunting): dump: 2024-03-25T18:07:45.932 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.932 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3632150038 0 0) 0x7fee5012b530 con 0x7fee500b4700 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee30004eb0 con 0x7fee500b4700 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fee38001b00 con 0x7fee500b4700 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.930+0000 7fee477fe640 10 monclient(hunting): dump: 2024-03-25T18:07:45.933 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.931+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1884279997 0 0) 0x7fee5012b3a0 con 0x7fee50142a20 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.931+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fee5012b530 con 0x7fee50142a20 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.931+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fee48001b50 con 0x7fee50142a20 2024-03-25T18:07:45.934 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.931+0000 7fee477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.931+0000 7fee477fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.931+0000 7fee477fe640 10 monclient(hunting): dump: 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:45.935 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:45.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1117108182 0 0) 0x7fee30004eb0 con 0x7fee500b4700 2024-03-25T18:07:45.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 >> v1:172.21.15.37:6790/0 conn(0x7fee500a96b0 legacy=0x7fee5012ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 >> v1:172.21.15.37:6789/0 conn(0x7fee50142a20 legacy=0x7fee501e1cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee477fe640 10 monclient: _finish_hunting 0 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee477fe640 1 monclient: found mon.b 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee477fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee5012cc10 con 0x7fee500b4700 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee477fe640 10 monclient: _finish_auth 0 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee477fe640 10 monclient: _check_auth_tickets 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee55754640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee55754640 10 monclient: _renew_subs 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee55754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee55754640 1 -- 172.21.15.117:0/1896123238 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee501d7f10 con 0x7fee500b4700 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee55754640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:45.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee55754640 10 monclient: _renew_subs 2024-03-25T18:07:45.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee55754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:45.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee55754640 1 -- 172.21.15.117:0/1896123238 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fee50132190 con 0x7fee500b4700 2024-03-25T18:07:45.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.933+0000 7fee55754640 1 librados: init done 2024-03-25T18:07:45.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.934+0000 7fee55754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:45.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.934+0000 7fee55754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:45.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.934+0000 7fee55754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:45.938 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:45.934+0000 7fee55754640 1 -- 172.21.15.117:0/1896123238 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee5012ce60 con 0x7fee500b4700 2024-03-25T18:07:45.943 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:45.942+0000 7f4033642040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:07:45.987 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:45.986+0000 7f4033642040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:07:46.000 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:45.999+0000 7f6850e56040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:07:46.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fee38001cc0 con 0x7fee500b4700 2024-03-25T18:07:46.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:46.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fee38003960 con 0x7fee500b4700 2024-03-25T18:07:46.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 10 monclient: dump: 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fee38003e50 con 0x7fee500b4700 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee4dffb640 4 set_mon_vals no callback set 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 1920600319 0 0) 0x7fee380053d0 con 0x7fee500b4700 2024-03-25T18:07:46.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 10 client.4141.objecter ms_dispatch 0x7fee50000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:07:46.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 3 client.4141.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:07:46.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 3 client.4141.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:07:46.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-25T18:07:46.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fee38005790 con 0x7fee500b4700 2024-03-25T18:07:46.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:46.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.034+0000 7fee477fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:46.099 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:46.099+0000 7f6850e56040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:07:46.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.112+0000 7fee55754640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d", "id": 4}] 2024-03-25T18:07:46.112 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.112+0000 7fee55754640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d", "id": 4}] 2024-03-25T18:07:46.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.112+0000 7fee55754640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:46.113 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.112+0000 7fee55754640 1 -- 172.21.15.117:0/1896123238 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d", "id": 4} v 0) v1 -- 0x7fee5004f730 con 0x7fee500b4700 2024-03-25T18:07:46.151 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:46.150+0000 7f4033642040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:07:46.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.174+0000 7fee477fe640 1 -- 172.21.15.117:0/1896123238 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 2790464928 0 1697012583) 0x7fee38004ec0 con 0x7fee500b4700 2024-03-25T18:07:46.174 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.174+0000 7fee477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d", "id": 4}] 2024-03-25T18:07:46.175 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.174+0000 7fee477fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:46.175 INFO:teuthology.orchestra.run.smithi117.stdout:4 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 10 librados: watch_flush enter 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 10 librados: watch_flush exit 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 10 client.4141.objecter successfully canceled tick 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 10 monclient: shutdown 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 1 -- 172.21.15.117:0/1896123238 >> v1:172.21.15.117:6789/0 conn(0x7fee500b4700 legacy=0x7fee501d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 1 -- 172.21.15.117:0/1896123238 shutdown_connections 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 1 -- 172.21.15.117:0/1896123238 >> 172.21.15.117:0/1896123238 conn(0x7fee500668e0 msgr2=0x7fee500b8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 1 -- 172.21.15.117:0/1896123238 shutdown_connections 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 1 -- 172.21.15.117:0/1896123238 wait complete. 2024-03-25T18:07:46.176 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.175+0000 7fee55754640 1 librados: shutdown 2024-03-25T18:07:46.186 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 028242bd-e284-45f2-bd64-e250bcaea3f9 5 2024-03-25T18:07:46.266 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:46.266+0000 7f6850e56040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:07:46.287 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.287+0000 7fd5692c8640 10 monclient: get_monmap_and_config 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.287+0000 7fd5692c8640 10 monclient: build_initial_monmap 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.287+0000 7fd5692c8640 10 monclient: monmap: 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:07:46.288308+0000 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:07:46.288308+0000 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:46.288 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.287+0000 7fd5692c8640 1 Processor -- start 2024-03-25T18:07:46.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.287+0000 7fd5692c8640 1 -- start start 2024-03-25T18:07:46.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.287+0000 7fd5692c8640 10 monclient: init 2024-03-25T18:07:46.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.288+0000 7fd5692c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:46.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.288+0000 7fd5692c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:46.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.288+0000 7fd5692c8640 10 monclient(hunting): picked mon.noname-a con 0x7fd5640b4700 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:46.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.288+0000 7fd5692c8640 10 monclient(hunting): picked mon.noname-b con 0x7fd564142a20 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:46.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.288+0000 7fd5692c8640 10 monclient(hunting): picked mon.noname-c con 0x7fd5640a96b0 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:46.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.288+0000 7fd5692c8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5640a9d60 con 0x7fd5640b4700 2024-03-25T18:07:46.290 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.288+0000 7fd5692c8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd56413ff10 con 0x7fd5640a96b0 2024-03-25T18:07:46.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.288+0000 7fd5692c8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5640aa1f0 con 0x7fd564142a20 2024-03-25T18:07:46.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.288+0000 7fd5692c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:46.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.288+0000 7fd5692c8640 10 monclient(hunting): authenticate will time out at 6821.418453s 2024-03-25T18:07:46.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.289+0000 7fd5627fc640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fd564142a20 0x7fd56413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.117:49028/0 (socket says 172.21.15.117:49028) 2024-03-25T18:07:46.291 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.289+0000 7fd5627fc640 1 -- 172.21.15.117:0/3222750137 learned_addr learned my addr 172.21.15.117:0/3222750137 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:46.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1457350703 0 0) 0x7fd5640a9d60 con 0x7fd5640b4700 2024-03-25T18:07:46.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:07:46.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd54c003520 con 0x7fd5640b4700 2024-03-25T18:07:46.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210321247 0 0) 0x7fd5640aa1f0 con 0x7fd564142a20 2024-03-25T18:07:46.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.309 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:07:46.310 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.308+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5640a9d60 con 0x7fd564142a20 2024-03-25T18:07:46.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.311+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608511890 0 0) 0x7fd56413ff10 con 0x7fd5640a96b0 2024-03-25T18:07:46.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.311+0000 7fd561ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.311+0000 7fd561ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.311+0000 7fd561ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.311+0000 7fd561ffb640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:07:46.312 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.311+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5640aa1f0 con 0x7fd5640a96b0 2024-03-25T18:07:46.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3493813362 0 0) 0x7fd5640a9d60 con 0x7fd564142a20 2024-03-25T18:07:46.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 >> v1:172.21.15.117:6789/0 conn(0x7fd5640a96b0 legacy=0x7fd56413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 >> v1:172.21.15.37:6789/0 conn(0x7fd5640b4700 legacy=0x7fd564142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.313 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 1 monclient: found mon.noname-b 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5640b4ae0 con 0x7fd564142a20 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 10 monclient: _finish_auth 0 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd558001670 con 0x7fd564142a20 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd5692c8640 5 monclient: authenticate success, global_id 4142 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd5692c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:46.314 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd561ffb640 10 monclient: dump: 2024-03-25T18:07:46.315 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.315 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.315 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.315 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.315 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.315 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.315 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.315 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.315 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.315 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.315 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.312+0000 7fd5692c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:46.318 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:46.317+0000 7f6850e56040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:07:46.340 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.340+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd5580017d0 con 0x7fd564142a20 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.340+0000 7fd561ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.340+0000 7fd561ffb640 1 -- 172.21.15.117:0/3222750137 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd558001be0 con 0x7fd564142a20 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.340+0000 7fd561ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.340+0000 7fd561ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.340+0000 7fd561ffb640 10 monclient: dump: 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.341 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.340+0000 7fd5692c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:46.342 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.340+0000 7fd5692c8640 4 set_mon_vals no callback set 2024-03-25T18:07:46.343 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.340+0000 7fd5692c8640 10 monclient: shutdown 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.340+0000 7fd5692c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.340+0000 7fd5692c8640 1 -- 172.21.15.117:0/3222750137 >> v1:172.21.15.37:6790/0 conn(0x7fd564142a20 legacy=0x7fd56413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.341+0000 7fd5692c8640 1 -- 172.21.15.117:0/3222750137 shutdown_connections 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.341+0000 7fd5692c8640 1 -- 172.21.15.117:0/3222750137 >> 172.21.15.117:0/3222750137 conn(0x7fd5640668e0 msgr2=0x7fd564066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.341+0000 7fd5692c8640 1 -- 172.21.15.117:0/3222750137 shutdown_connections 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.341+0000 7fd5692c8640 1 -- 172.21.15.117:0/3222750137 wait complete. 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.341+0000 7fd5692c8640 10 monclient: build_initial_monmap 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.341+0000 7fd5692c8640 10 monclient: monmap: 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:46.344 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 1 librados: starting msgr at 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 1 librados: starting objecter 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 1 Processor -- start 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 1 -- start start 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 1 librados: setting wanted keys 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 1 librados: calling monclient init 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 10 monclient: init 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:07:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 10 monclient(hunting): picked mon.noname-b con 0x7fd5640a96b0 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 10 monclient(hunting): picked mon.noname-a con 0x7fd5640b4700 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 10 monclient(hunting): picked mon.noname-c con 0x7fd564142a20 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5641d7de0 con 0x7fd5640b4700 2024-03-25T18:07:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5640ad890 con 0x7fd5640a96b0 2024-03-25T18:07:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5641d80a0 con 0x7fd564142a20 2024-03-25T18:07:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.342+0000 7fd5692c8640 10 monclient(hunting): authenticate will time out at 6821.472771s 2024-03-25T18:07:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.343+0000 7fd5627fc640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd5640b4700 0x7fd5640b0530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.117:56558/0 (socket says 172.21.15.117:56558) 2024-03-25T18:07:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.343+0000 7fd5637fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fd564142a20 0x7fd5641d3fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.117:49036/0 (socket says 172.21.15.117:49036) 2024-03-25T18:07:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.343+0000 7fd562ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd5640a96b0 0x7fd5641d76d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58924/0 (socket says 172.21.15.117:58924) 2024-03-25T18:07:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.343+0000 7fd5627fc640 1 -- 172.21.15.117:0/789568152 learned_addr learned my addr 172.21.15.117:0/789568152 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:46.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.343+0000 7fd54b7fe640 10 client.?.objecter ms_handle_connect 0x7fd564142a20 2024-03-25T18:07:46.347 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.343+0000 7fd54b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.343+0000 7fd54b7fe640 10 client.?.objecter ms_handle_connect 0x7fd5640a96b0 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.343+0000 7fd54b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.343+0000 7fd54b7fe640 10 client.?.objecter ms_handle_connect 0x7fd5640b4700 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.343+0000 7fd54b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.345+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3595697628 0 0) 0x7fd5640ad890 con 0x7fd5640a96b0 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.345+0000 7fd54b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.345+0000 7fd54b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.345+0000 7fd54b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.345+0000 7fd54b7fe640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.345+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd550004eb0 con 0x7fd5640a96b0 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466162292 0 0) 0x7fd5641d80a0 con 0x7fd564142a20 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:07:46.348 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5640ad890 con 0x7fd564142a20 2024-03-25T18:07:46.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 561220092 0 0) 0x7fd5641d7de0 con 0x7fd5640b4700 2024-03-25T18:07:46.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:07:46.349 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.346+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5641d80a0 con 0x7fd5640b4700 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.350+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1831193388 0 0) 0x7fd550004eb0 con 0x7fd5640a96b0 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.350+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5641d7de0 con 0x7fd5640a96b0 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.350+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd5540018d0 con 0x7fd5640a96b0 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.350+0000 7fd54b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.350+0000 7fd54b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.350+0000 7fd54b7fe640 10 monclient(hunting): dump: 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.351 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.352 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.352 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.352 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.352 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.352 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.350+0000 7fd54b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:46.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 530572016 0 0) 0x7fd5640ad890 con 0x7fd564142a20 2024-03-25T18:07:46.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd550004eb0 con 0x7fd564142a20 2024-03-25T18:07:46.353 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd55c001b50 con 0x7fd564142a20 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 10 monclient(hunting): dump: 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.354 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1490215087 0 0) 0x7fd5641d80a0 con 0x7fd5640b4700 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5640ad890 con 0x7fd5640b4700 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd558001b60 con 0x7fd5640b4700 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.352+0000 7fd54b7fe640 10 monclient(hunting): dump: 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.355 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.356 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2220892413 0 0) 0x7fd5641d7de0 con 0x7fd5640a96b0 2024-03-25T18:07:46.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 >> v1:172.21.15.37:6790/0 conn(0x7fd564142a20 legacy=0x7fd5641d3fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 >> v1:172.21.15.37:6789/0 conn(0x7fd5640b4700 legacy=0x7fd5640b0530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd54b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:07:46.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd54b7fe640 1 monclient: found mon.b 2024-03-25T18:07:46.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd54b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:46.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd54b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:46.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5640adae0 con 0x7fd5640a96b0 2024-03-25T18:07:46.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd54b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:07:46.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd54b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:07:46.357 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd54b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd5692c8640 5 monclient: authenticate success, global_id 4147 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd5692c8640 10 monclient: _renew_subs 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd5692c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd5692c8640 1 -- 172.21.15.117:0/789568152 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5640b1790 con 0x7fd5640a96b0 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd5692c8640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd5692c8640 10 monclient: _renew_subs 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd5692c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd5692c8640 1 -- 172.21.15.117:0/789568152 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5640b0e20 con 0x7fd5640a96b0 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.353+0000 7fd5692c8640 1 librados: init done 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.354+0000 7fd5692c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.354+0000 7fd5692c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.354+0000 7fd5692c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:46.358 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.354+0000 7fd5692c8640 1 -- 172.21.15.117:0/789568152 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5640b1ae0 con 0x7fd5640a96b0 2024-03-25T18:07:46.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd554003610 con 0x7fd5640a96b0 2024-03-25T18:07:46.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:46.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd5540038c0 con 0x7fd5640a96b0 2024-03-25T18:07:46.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:46.361 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 10 monclient: dump: 2024-03-25T18:07:46.362 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.362 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.362 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.362 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.362 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.362 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.362 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.362 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.362 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.362 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.362 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd554003e70 con 0x7fd5640a96b0 2024-03-25T18:07:46.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2099083278 0 0) 0x7fd554006600 con 0x7fd5640a96b0 2024-03-25T18:07:46.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 10 client.4147.objecter ms_dispatch 0x7fd564000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:07:46.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd5617fa640 4 set_mon_vals no callback set 2024-03-25T18:07:46.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 3 client.4147.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:07:46.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 3 client.4147.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:07:46.363 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.357+0000 7fd54b7fe640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T18:07:46.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.450+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd5540057c0 con 0x7fd5640a96b0 2024-03-25T18:07:46.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.450+0000 7fd54b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:46.451 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.450+0000 7fd54b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:46.480 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:46.479+0000 7f6850e56040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:07:46.523 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.522+0000 7fd5692c8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "028242bd-e284-45f2-bd64-e250bcaea3f9", "id": 5}] 2024-03-25T18:07:46.523 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.522+0000 7fd5692c8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "028242bd-e284-45f2-bd64-e250bcaea3f9", "id": 5}] 2024-03-25T18:07:46.523 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.522+0000 7fd5692c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:46.523 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.522+0000 7fd5692c8640 1 -- 172.21.15.117:0/789568152 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd new", "uuid": "028242bd-e284-45f2-bd64-e250bcaea3f9", "id": 5} v 0) v1 -- 0x7fd56404f730 con 0x7fd5640a96b0 2024-03-25T18:07:46.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.600+0000 7fd54b7fe640 1 -- 172.21.15.117:0/789568152 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "028242bd-e284-45f2-bd64-e250bcaea3f9", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3619592483 0 1988412176) 0x7fd554005960 con 0x7fd5640a96b0 2024-03-25T18:07:46.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.600+0000 7fd54b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "028242bd-e284-45f2-bd64-e250bcaea3f9", "id": 5}] 2024-03-25T18:07:46.601 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.600+0000 7fd54b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:46.601 INFO:teuthology.orchestra.run.smithi117.stdout:5 2024-03-25T18:07:46.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 10 librados: watch_flush enter 2024-03-25T18:07:46.602 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 10 librados: watch_flush exit 2024-03-25T18:07:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T18:07:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 10 client.4147.objecter successfully canceled tick 2024-03-25T18:07:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 10 monclient: shutdown 2024-03-25T18:07:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 1 -- 172.21.15.117:0/789568152 >> v1:172.21.15.117:6789/0 conn(0x7fd5640a96b0 legacy=0x7fd5641d76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 1 -- 172.21.15.117:0/789568152 shutdown_connections 2024-03-25T18:07:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 1 -- 172.21.15.117:0/789568152 >> 172.21.15.117:0/789568152 conn(0x7fd5640668e0 msgr2=0x7fd5640b6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 1 -- 172.21.15.117:0/789568152 shutdown_connections 2024-03-25T18:07:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 1 -- 172.21.15.117:0/789568152 wait complete. 2024-03-25T18:07:46.603 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.601+0000 7fd5692c8640 1 librados: shutdown 2024-03-25T18:07:46.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new b9f96f7d-77ce-40be-a2eb-041c13198fbf 6 2024-03-25T18:07:46.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.709+0000 7f691be95640 10 monclient: get_monmap_and_config 2024-03-25T18:07:46.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.709+0000 7f691be95640 10 monclient: build_initial_monmap 2024-03-25T18:07:46.710 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.709+0000 7f691be95640 10 monclient: monmap: 2024-03-25T18:07:46.710 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:46.710 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.710 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:07:46.710507+0000 2024-03-25T18:07:46.710 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:07:46.710507+0000 2024-03-25T18:07:46.710 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:46.710 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.710 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:46.710 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:46.711 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:46.711 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 1 Processor -- start 2024-03-25T18:07:46.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 1 -- start start 2024-03-25T18:07:46.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 10 monclient: init 2024-03-25T18:07:46.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:46.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:46.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 10 monclient(hunting): picked mon.noname-b con 0x7f69140a9fa0 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:46.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 10 monclient(hunting): picked mon.noname-c con 0x7f69140ab520 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:46.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 10 monclient(hunting): picked mon.noname-a con 0x7f69140afc70 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:46.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69140ac500 con 0x7f69140afc70 2024-03-25T18:07:46.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69140ab040 con 0x7f69140ab520 2024-03-25T18:07:46.712 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6914072a30 con 0x7f69140a9fa0 2024-03-25T18:07:46.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:46.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.710+0000 7f691be95640 10 monclient(hunting): authenticate will time out at 6821.840741s 2024-03-25T18:07:46.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.711+0000 7f6919430640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f69140ab520 0x7f691413bbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58940/0 (socket says 172.21.15.117:58940) 2024-03-25T18:07:46.713 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.711+0000 7f6919430640 1 -- 172.21.15.117:0/152157904 learned_addr learned my addr 172.21.15.117:0/152157904 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:46.714 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.713+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1688238532 0 0) 0x7f69140ab040 con 0x7f69140ab520 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.713+0000 7f6918c2f640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.713+0000 7f6918c2f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.713+0000 7f6918c2f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.713+0000 7f6918c2f640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.713+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6900004630 con 0x7f69140ab520 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164026360 0 0) 0x7f6914072a30 con 0x7f69140a9fa0 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69140ab040 con 0x7f69140a9fa0 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590869604 0 0) 0x7f69140ac500 con 0x7f69140afc70 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.715 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:07:46.716 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.714+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6914072a30 con 0x7f69140afc70 2024-03-25T18:07:46.717 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2924242351 0 0) 0x7f6900004630 con 0x7f69140ab520 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 >> v1:172.21.15.37:6790/0 conn(0x7f69140a9fa0 legacy=0x7f69140af560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 >> v1:172.21.15.37:6789/0 conn(0x7f69140afc70 legacy=0x7f691414a260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 10 monclient: _finish_hunting 0 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 1 monclient: found mon.noname-c 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69140ac770 con 0x7f69140ab520 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 10 monclient: _finish_auth 0 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 10 monclient: _check_auth_tickets 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6910001690 con 0x7f69140ab520 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:46.718 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f6918c2f640 10 monclient: dump: 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f691be95640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:07:46.719 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.716+0000 7f691be95640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:46.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.718+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69100017f0 con 0x7f69140ab520 2024-03-25T18:07:46.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.718+0000 7f6918c2f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:46.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.718+0000 7f6918c2f640 1 -- 172.21.15.117:0/152157904 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6910001b80 con 0x7f69140ab520 2024-03-25T18:07:46.722 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.718+0000 7f6918c2f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.718+0000 7f6918c2f640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.718+0000 7f6918c2f640 10 monclient: dump: 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.718+0000 7f691be95640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.718+0000 7f691be95640 4 set_mon_vals no callback set 2024-03-25T18:07:46.723 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.719+0000 7f691be95640 10 monclient: shutdown 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.719+0000 7f691be95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.719+0000 7f691be95640 1 -- 172.21.15.117:0/152157904 >> v1:172.21.15.117:6789/0 conn(0x7f69140ab520 legacy=0x7f691413bbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.719+0000 7f691be95640 1 -- 172.21.15.117:0/152157904 shutdown_connections 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.719+0000 7f691be95640 1 -- 172.21.15.117:0/152157904 >> 172.21.15.117:0/152157904 conn(0x7f691406b560 msgr2=0x7f69140b3aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.719+0000 7f691be95640 1 -- 172.21.15.117:0/152157904 shutdown_connections 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.719+0000 7f691be95640 1 -- 172.21.15.117:0/152157904 wait complete. 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.719+0000 7f691be95640 10 monclient: build_initial_monmap 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.719+0000 7f691be95640 10 monclient: monmap: 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-25T18:07:46.724 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 1 librados: starting msgr at 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 1 librados: starting objecter 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 1 Processor -- start 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 1 -- start start 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 1 librados: setting wanted keys 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 1 librados: calling monclient init 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 10 monclient: init 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 10 monclient(hunting): picked mon.noname-c con 0x7f69140a9fa0 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 10 monclient(hunting): picked mon.noname-a con 0x7f69140afc70 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 10 monclient(hunting): picked mon.noname-b con 0x7f6914135120 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:46.725 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69141ca280 con 0x7f69140afc70 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69141ca5a0 con 0x7f6914135120 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69141db5f0 con 0x7f69140a9fa0 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.720+0000 7f691be95640 10 monclient(hunting): authenticate will time out at 6821.850813s 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.721+0000 7f691a432640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f6914135120 0x7f69141daee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58954/0 (socket says 172.21.15.117:58954) 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.721+0000 7f691a432640 1 -- 172.21.15.117:0/3265007765 learned_addr learned my addr 172.21.15.117:0/3265007765 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.721+0000 7f69067fc640 10 client.?.objecter ms_handle_connect 0x7f6914135120 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.721+0000 7f69067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.721+0000 7f69067fc640 10 client.?.objecter ms_handle_connect 0x7f69140a9fa0 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.721+0000 7f69067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.721+0000 7f69067fc640 10 client.?.objecter ms_handle_connect 0x7f69140afc70 2024-03-25T18:07:46.726 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.721+0000 7f69067fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:46.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.723+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 685898179 0 0) 0x7f69141db5f0 con 0x7f69140a9fa0 2024-03-25T18:07:46.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.723+0000 7f69067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.723+0000 7f69067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.723+0000 7f69067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.727 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.723+0000 7f69067fc640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.723+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68f8004eb0 con 0x7f69140a9fa0 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.724+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1291042610 0 0) 0x7f69141ca280 con 0x7f69140afc70 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.724+0000 7f69067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.724+0000 7f69067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.724+0000 7f69067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.724+0000 7f69067fc640 10 monclient(hunting): my global_id is 4152 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.724+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69141db5f0 con 0x7f69140afc70 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.725+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944504011 0 0) 0x7f69141ca5a0 con 0x7f6914135120 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.725+0000 7f69067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.725+0000 7f69067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.725+0000 7f69067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.725+0000 7f69067fc640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:07:46.728 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.725+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f69141ca280 con 0x7f6914135120 2024-03-25T18:07:46.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.728+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3609284577 0 0) 0x7f68f8004eb0 con 0x7f69140a9fa0 2024-03-25T18:07:46.729 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.728+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69141ca5a0 con 0x7f69140a9fa0 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.729+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6908001a10 con 0x7f69140a9fa0 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.729+0000 7f69067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.729+0000 7f69067fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.729+0000 7f69067fc640 10 monclient(hunting): dump: 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.730 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.729+0000 7f69067fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:46.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3239895660 0 0) 0x7f69141db5f0 con 0x7f69140afc70 2024-03-25T18:07:46.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68f8004eb0 con 0x7f69140afc70 2024-03-25T18:07:46.731 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6910001960 con 0x7f69140afc70 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 10 monclient(hunting): dump: 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.732 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3823924038 0 0) 0x7f69141ca280 con 0x7f6914135120 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f69141db5f0 con 0x7f6914135120 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f690c001bd0 con 0x7f6914135120 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.730+0000 7f69067fc640 10 monclient(hunting): dump: 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.733 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3024321160 0 0) 0x7f69141ca5a0 con 0x7f69140a9fa0 2024-03-25T18:07:46.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 >> v1:172.21.15.117:6789/0 conn(0x7f6914135120 legacy=0x7f69141daee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 >> v1:172.21.15.37:6789/0 conn(0x7f69140afc70 legacy=0x7f69141c9b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:46.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f69067fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:46.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f69067fc640 1 monclient: found mon.c 2024-03-25T18:07:46.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f69067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:46.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f69067fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:07:46.735 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69141db780 con 0x7f69140a9fa0 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f69067fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f69067fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f69067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f691be95640 5 monclient: authenticate success, global_id 4151 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f691be95640 10 monclient: _renew_subs 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f691be95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f691be95640 1 -- 172.21.15.117:0/3265007765 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69141dba30 con 0x7f69140a9fa0 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f691be95640 10 client.4151.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f691be95640 10 monclient: _renew_subs 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f691be95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f691be95640 1 -- 172.21.15.117:0/3265007765 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6914135790 con 0x7f69140a9fa0 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.732+0000 7f691be95640 1 librados: init done 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.733+0000 7f691be95640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.733+0000 7f691be95640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.733+0000 7f691be95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:07:46.736 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.733+0000 7f691be95640 1 -- 172.21.15.117:0/3265007765 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6914135ae0 con 0x7f69140a9fa0 2024-03-25T18:07:46.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.737+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f69080035f0 con 0x7f69140a9fa0 2024-03-25T18:07:46.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.737+0000 7f69067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:46.738 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.738+0000 7f6907fff640 4 set_mon_vals no callback set 2024-03-25T18:07:46.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.935+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6908003900 con 0x7f69140a9fa0 2024-03-25T18:07:46.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.935+0000 7f69067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:46.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.935+0000 7f69067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:46.936 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.935+0000 7f69067fc640 10 monclient: dump: 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.935+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f6908003e30 con 0x7f69140a9fa0 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.935+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 2219099892 0 0) 0x7f69080056f0 con 0x7f69140a9fa0 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.935+0000 7f69067fc640 10 client.4151.objecter ms_dispatch 0x7f6914000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.935+0000 7f69067fc640 3 client.4151.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.935+0000 7f69067fc640 3 client.4151.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:07:46.937 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.935+0000 7f69067fc640 20 client.4151.objecter dump_active .. 0 homeless 2024-03-25T18:07:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.938+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6908005210 con 0x7f69140a9fa0 2024-03-25T18:07:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.938+0000 7f69067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:46.939 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:46.938+0000 7f69067fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:47.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.032+0000 7f691be95640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "b9f96f7d-77ce-40be-a2eb-041c13198fbf", "id": 6}] 2024-03-25T18:07:47.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.032+0000 7f691be95640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "b9f96f7d-77ce-40be-a2eb-041c13198fbf", "id": 6}] 2024-03-25T18:07:47.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.032+0000 7f691be95640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:07:47.033 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.032+0000 7f691be95640 1 -- 172.21.15.117:0/3265007765 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd new", "uuid": "b9f96f7d-77ce-40be-a2eb-041c13198fbf", "id": 6} v 0) v1 -- 0x7f69141cb8c0 con 0x7f69140a9fa0 2024-03-25T18:07:47.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.149+0000 7f69067fc640 1 -- 172.21.15.117:0/3265007765 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b9f96f7d-77ce-40be-a2eb-041c13198fbf", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 3478884952 0 1113811849) 0x7f6908005500 con 0x7f69140a9fa0 2024-03-25T18:07:47.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.149+0000 7f69067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "b9f96f7d-77ce-40be-a2eb-041c13198fbf", "id": 6}] 2024-03-25T18:07:47.150 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.149+0000 7f69067fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:47.150 INFO:teuthology.orchestra.run.smithi117.stdout:6 2024-03-25T18:07:47.151 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 10 librados: watch_flush enter 2024-03-25T18:07:47.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 10 librados: watch_flush exit 2024-03-25T18:07:47.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 20 client.4151.objecter shutdown clearing up homeless session... 2024-03-25T18:07:47.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 10 client.4151.objecter successfully canceled tick 2024-03-25T18:07:47.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 10 monclient: shutdown 2024-03-25T18:07:47.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:47.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 1 -- 172.21.15.117:0/3265007765 >> v1:172.21.15.37:6790/0 conn(0x7f69140a9fa0 legacy=0x7f69141c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 1 -- 172.21.15.117:0/3265007765 shutdown_connections 2024-03-25T18:07:47.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 1 -- 172.21.15.117:0/3265007765 >> 172.21.15.117:0/3265007765 conn(0x7f691406b560 msgr2=0x7f69140b2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:47.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 1 -- 172.21.15.117:0/3265007765 shutdown_connections 2024-03-25T18:07:47.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 1 -- 172.21.15.117:0/3265007765 wait complete. 2024-03-25T18:07:47.152 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.150+0000 7f691be95640 1 librados: shutdown 2024-03-25T18:07:47.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 57e48c8b-9e7c-4113-9d08-0c858b9faf5b 7 2024-03-25T18:07:47.232 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:47.231+0000 7f4033642040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:07:47.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.259+0000 7f8f4b19d640 10 monclient: get_monmap_and_config 2024-03-25T18:07:47.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.259+0000 7f8f4b19d640 10 monclient: build_initial_monmap 2024-03-25T18:07:47.260 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.259+0000 7f8f4b19d640 10 monclient: monmap: 2024-03-25T18:07:47.260 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:47.260 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.261 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:07:47.260915+0000 2024-03-25T18:07:47.261 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:07:47.260915+0000 2024-03-25T18:07:47.261 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:47.261 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:47.261 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:47.261 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:47.261 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:47.261 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:47.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.260+0000 7f8f4b19d640 1 Processor -- start 2024-03-25T18:07:47.262 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.260+0000 7f8f4b19d640 1 -- start start 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.260+0000 7f8f4b19d640 10 monclient: init 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f4b19d640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f4b19d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f4b19d640 10 monclient(hunting): picked mon.noname-a con 0x7f8f440b4700 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f4b19d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f44142a20 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f4b19d640 10 monclient(hunting): picked mon.noname-b con 0x7f8f440a96b0 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f4b19d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f440a9d60 con 0x7f8f440b4700 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f4b19d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f4413ff10 con 0x7f8f44142a20 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f4b19d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f440aa1f0 con 0x7f8f440a96b0 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f4b19d640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f4b19d640 10 monclient(hunting): authenticate will time out at 6822.391172s 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f43fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f8f44142a20 0x7f8f4413f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58956/0 (socket says 172.21.15.117:58956) 2024-03-25T18:07:47.263 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.261+0000 7f8f43fff640 1 -- 172.21.15.117:0/994702635 learned_addr learned my addr 172.21.15.117:0/994702635 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:47.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440401586 0 0) 0x7f8f4413ff10 con 0x7f8f44142a20 2024-03-25T18:07:47.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): my global_id is 4156 2024-03-25T18:07:47.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f34003520 con 0x7f8f44142a20 2024-03-25T18:07:47.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1551744935 0 0) 0x7f8f440a9d60 con 0x7f8f440b4700 2024-03-25T18:07:47.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.265 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:07:47.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f4413ff10 con 0x7f8f440b4700 2024-03-25T18:07:47.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647421373 0 0) 0x7f8f440aa1f0 con 0x7f8f440a96b0 2024-03-25T18:07:47.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:07:47.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.264+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f440a9d60 con 0x7f8f440a96b0 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 973501617 0 0) 0x7f8f34003520 con 0x7f8f44142a20 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 >> v1:172.21.15.37:6790/0 conn(0x7f8f440a96b0 legacy=0x7f8f4413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 >> v1:172.21.15.37:6789/0 conn(0x7f8f440b4700 legacy=0x7f8f44142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 10 monclient: _finish_hunting 0 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 1 monclient: found mon.noname-c 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f440b4ae0 con 0x7f8f44142a20 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 10 monclient: _finish_auth 0 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 10 monclient: _check_auth_tickets 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:47.268 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8f30001690 con 0x7f8f44142a20 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f4b19d640 5 monclient: authenticate success, global_id 4156 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f4b19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f437fe640 10 monclient: dump: 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:47.269 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:47.270 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:47.270 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.267+0000 7f8f4b19d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:47.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.268+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f300017f0 con 0x7f8f44142a20 2024-03-25T18:07:47.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.268+0000 7f8f437fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:47.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.268+0000 7f8f437fe640 1 -- 172.21.15.117:0/994702635 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8f30001c00 con 0x7f8f44142a20 2024-03-25T18:07:47.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.268+0000 7f8f4b19d640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:47.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.268+0000 7f8f4b19d640 4 set_mon_vals no callback set 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.268+0000 7f8f437fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.268+0000 7f8f437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.268+0000 7f8f437fe640 10 monclient: dump: 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.269+0000 7f8f4b19d640 10 monclient: shutdown 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.269+0000 7f8f4b19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.269+0000 7f8f4b19d640 1 -- 172.21.15.117:0/994702635 >> v1:172.21.15.117:6789/0 conn(0x7f8f44142a20 legacy=0x7f8f4413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.269+0000 7f8f4b19d640 1 -- 172.21.15.117:0/994702635 shutdown_connections 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.269+0000 7f8f4b19d640 1 -- 172.21.15.117:0/994702635 >> 172.21.15.117:0/994702635 conn(0x7f8f440668e0 msgr2=0x7f8f44066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.269+0000 7f8f4b19d640 1 -- 172.21.15.117:0/994702635 shutdown_connections 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.269+0000 7f8f4b19d640 1 -- 172.21.15.117:0/994702635 wait complete. 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.270+0000 7f8f4b19d640 10 monclient: build_initial_monmap 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.270+0000 7f8f4b19d640 10 monclient: monmap: 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 0 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 0.000000 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:created 0.000000 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:47.273 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 1 2024-03-25T18:07:47.274 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:47.274 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:47.274 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:47.274 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:47.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.270+0000 7f8f4b19d640 1 librados: starting msgr at 2024-03-25T18:07:47.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.270+0000 7f8f4b19d640 1 librados: starting objecter 2024-03-25T18:07:47.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.270+0000 7f8f4b19d640 1 Processor -- start 2024-03-25T18:07:47.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.270+0000 7f8f4b19d640 1 -- start start 2024-03-25T18:07:47.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.270+0000 7f8f4b19d640 1 librados: setting wanted keys 2024-03-25T18:07:47.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.270+0000 7f8f4b19d640 1 librados: calling monclient init 2024-03-25T18:07:47.274 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.270+0000 7f8f4b19d640 10 monclient: init 2024-03-25T18:07:47.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.271+0000 7f8f4b19d640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:47.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.271+0000 7f8f4b19d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:47.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.271+0000 7f8f4b19d640 10 monclient(hunting): picked mon.noname-c con 0x7f8f440a96b0 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:47.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.271+0000 7f8f4b19d640 10 monclient(hunting): picked mon.noname-b con 0x7f8f440b4700 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:47.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.271+0000 7f8f4b19d640 10 monclient(hunting): picked mon.noname-a con 0x7f8f44142a20 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:47.276 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.271+0000 7f8f4b19d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f440ae010 con 0x7f8f44142a20 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.271+0000 7f8f4b19d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f440ae2f0 con 0x7f8f440b4700 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.271+0000 7f8f4b19d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f440af100 con 0x7f8f440a96b0 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.271+0000 7f8f4b19d640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.271+0000 7f8f4b19d640 10 monclient(hunting): authenticate will time out at 6822.401463s 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.272+0000 7f8f43fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f8f440b4700 0x7f8f44139e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:58964/0 (socket says 172.21.15.117:58964) 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.272+0000 7f8f43fff640 1 -- 172.21.15.117:0/1065609075 learned_addr learned my addr 172.21.15.117:0/1065609075 (peer_addr_for_me v1:172.21.15.117:0/0) 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.272+0000 7f8f417fa640 10 client.?.objecter ms_handle_connect 0x7f8f440b4700 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.272+0000 7f8f417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.272+0000 7f8f417fa640 10 client.?.objecter ms_handle_connect 0x7f8f440a96b0 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.272+0000 7f8f417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3047479276 0 0) 0x7f8f440ae2f0 con 0x7f8f440b4700 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.277 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 10 monclient(hunting): my global_id is 4159 2024-03-25T18:07:47.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f28004eb0 con 0x7f8f440b4700 2024-03-25T18:07:47.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1519014485 0 0) 0x7f8f440af100 con 0x7f8f440a96b0 2024-03-25T18:07:47.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:07:47.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.274+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f440ae2f0 con 0x7f8f440a96b0 2024-03-25T18:07:47.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.275+0000 7f8f417fa640 10 client.?.objecter ms_handle_connect 0x7f8f44142a20 2024-03-25T18:07:47.278 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.275+0000 7f8f417fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:47.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.278+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4183028983 0 0) 0x7f8f440ae010 con 0x7f8f44142a20 2024-03-25T18:07:47.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.278+0000 7f8f417fa640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.278+0000 7f8f417fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.278+0000 7f8f417fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.278+0000 7f8f417fa640 10 monclient(hunting): my global_id is 4158 2024-03-25T18:07:47.279 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.278+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f440af100 con 0x7f8f44142a20 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.279+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3871760649 0 0) 0x7f8f28004eb0 con 0x7f8f440b4700 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.279+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f440ae010 con 0x7f8f440b4700 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.279+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8f30001bc0 con 0x7f8f440b4700 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.279+0000 7f8f417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.279+0000 7f8f417fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.279+0000 7f8f417fa640 10 monclient(hunting): dump: 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:47.280 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:47.281 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:47.281 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.279+0000 7f8f417fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:47.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.282+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4195375939 0 0) 0x7f8f440af100 con 0x7f8f44142a20 2024-03-25T18:07:47.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.282+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f28004eb0 con 0x7f8f44142a20 2024-03-25T18:07:47.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.282+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8f3c001bd0 con 0x7f8f44142a20 2024-03-25T18:07:47.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.282+0000 7f8f417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:47.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.282+0000 7f8f417fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:47.283 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.282+0000 7f8f417fa640 10 monclient(hunting): dump: 2024-03-25T18:07:47.283 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:47.283 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.284 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.284 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.284 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:47.284 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:47.284 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:47.284 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:47.284 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:47.284 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:47.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3588167980 0 0) 0x7f8f28004eb0 con 0x7f8f44142a20 2024-03-25T18:07:47.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 >> v1:172.21.15.37:6790/0 conn(0x7f8f440a96b0 legacy=0x7f8f440ae9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 >> v1:172.21.15.117:6789/0 conn(0x7f8f440b4700 legacy=0x7f8f44139e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f417fa640 10 monclient: _finish_hunting 0 2024-03-25T18:07:47.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f417fa640 1 monclient: found mon.a 2024-03-25T18:07:47.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:47.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f417fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:47.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f440ad120 con 0x7f8f44142a20 2024-03-25T18:07:47.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f417fa640 10 monclient: _finish_auth 0 2024-03-25T18:07:47.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f417fa640 10 monclient: _check_auth_tickets 2024-03-25T18:07:47.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:47.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f4b19d640 5 monclient: authenticate success, global_id 4158 2024-03-25T18:07:47.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f4b19d640 10 monclient: _renew_subs 2024-03-25T18:07:47.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f4b19d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:47.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f4b19d640 1 -- 172.21.15.117:0/1065609075 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f440ad3f0 con 0x7f8f44142a20 2024-03-25T18:07:47.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f4b19d640 10 client.4158.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:47.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f4b19d640 10 monclient: _renew_subs 2024-03-25T18:07:47.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f4b19d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:47.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f4b19d640 1 -- 172.21.15.117:0/1065609075 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f440ac9c0 con 0x7f8f44142a20 2024-03-25T18:07:47.295 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.293+0000 7f8f4b19d640 1 librados: init done 2024-03-25T18:07:47.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.294+0000 7f8f4b19d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:47.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.294+0000 7f8f4b19d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:47.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.294+0000 7f8f4b19d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:47.296 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.294+0000 7f8f4b19d640 1 -- 172.21.15.117:0/1065609075 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f4404f730 con 0x7f8f44142a20 2024-03-25T18:07:47.343 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:47.342+0000 7f6850e56040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:07:47.426 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f3c001310 con 0x7f8f44142a20 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8f3c003990 con 0x7f8f44142a20 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 10 monclient: dump: 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:epoch 1 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:election_strategy: 3 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:47.428 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:07:47.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8f3c003f00 con 0x7f8f44142a20 2024-03-25T18:07:47.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f42ffd640 4 set_mon_vals no callback set 2024-03-25T18:07:47.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2567863340 0 0) 0x7f8f3c0057e0 con 0x7f8f44142a20 2024-03-25T18:07:47.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 10 client.4158.objecter ms_dispatch 0x7f8f44000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:07:47.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 3 client.4158.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:07:47.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 3 client.4158.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:07:47.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.425+0000 7f8f417fa640 20 client.4158.objecter dump_active .. 0 homeless 2024-03-25T18:07:47.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.427+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8f3c005290 con 0x7f8f44142a20 2024-03-25T18:07:47.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.427+0000 7f8f417fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:47.428 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.427+0000 7f8f417fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:47.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.499+0000 7f8f4b19d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "57e48c8b-9e7c-4113-9d08-0c858b9faf5b", "id": 7}] 2024-03-25T18:07:47.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.499+0000 7f8f4b19d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "57e48c8b-9e7c-4113-9d08-0c858b9faf5b", "id": 7}] 2024-03-25T18:07:47.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.499+0000 7f8f4b19d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:47.499 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.499+0000 7f8f4b19d640 1 -- 172.21.15.117:0/1065609075 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd new", "uuid": "57e48c8b-9e7c-4113-9d08-0c858b9faf5b", "id": 7} v 0) v1 -- 0x7f8f44142220 con 0x7f8f44142a20 2024-03-25T18:07:47.512 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:47.511+0000 7f4033642040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:07:47.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.530+0000 7f8f417fa640 1 -- 172.21.15.117:0/1065609075 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "57e48c8b-9e7c-4113-9d08-0c858b9faf5b", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 4132881467 0 1371665406) 0x7f8f3c005590 con 0x7f8f44142a20 2024-03-25T18:07:47.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.530+0000 7f8f417fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "57e48c8b-9e7c-4113-9d08-0c858b9faf5b", "id": 7}] 2024-03-25T18:07:47.531 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.530+0000 7f8f417fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:47.531 INFO:teuthology.orchestra.run.smithi117.stdout:7 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.531+0000 7f8f4b19d640 10 librados: watch_flush enter 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.531+0000 7f8f4b19d640 10 librados: watch_flush exit 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.531+0000 7f8f4b19d640 20 client.4158.objecter shutdown clearing up homeless session... 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.531+0000 7f8f4b19d640 10 client.4158.objecter successfully canceled tick 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.531+0000 7f8f4b19d640 10 monclient: shutdown 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.531+0000 7f8f4b19d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.531+0000 7f8f4b19d640 1 -- 172.21.15.117:0/1065609075 >> v1:172.21.15.37:6789/0 conn(0x7f8f44142a20 legacy=0x7f8f4413a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.531+0000 7f8f4b19d640 1 -- 172.21.15.117:0/1065609075 shutdown_connections 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.531+0000 7f8f4b19d640 1 -- 172.21.15.117:0/1065609075 >> 172.21.15.117:0/1065609075 conn(0x7f8f440668e0 msgr2=0x7f8f4413c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.531+0000 7f8f4b19d640 1 -- 172.21.15.117:0/1065609075 shutdown_connections 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.531+0000 7f8f4b19d640 1 -- 172.21.15.117:0/1065609075 wait complete. 2024-03-25T18:07:47.533 INFO:teuthology.orchestra.run.smithi117.stderr:2024-03-25T18:07:47.532+0000 7f8f4b19d640 1 librados: shutdown 2024-03-25T18:07:47.542 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:07:47.542 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:07:47.543 DEBUG:teuthology.orchestra.run.smithi037:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:07:47.545 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:07:47.545 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:07:47.545 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:07:47.545 DEBUG:teuthology.orchestra.run.smithi037:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:07:47.547 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:07:47.547 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:07:47.547 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:07:47.548 DEBUG:teuthology.orchestra.run.smithi037:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:07:47.550 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:07:47.550 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:07:47.550 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:07:47.550 DEBUG:teuthology.orchestra.run.smithi037:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:07:47.553 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:07:47.553 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:07:47.553 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:07:47.553 DEBUG:teuthology.orchestra.run.smithi117:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:07:47.555 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:07:47.555 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:07:47.556 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:07:47.556 DEBUG:teuthology.orchestra.run.smithi117:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:07:47.558 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:07:47.558 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:07:47.558 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:07:47.558 DEBUG:teuthology.orchestra.run.smithi117:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:07:47.560 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:07:47.560 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:07:47.560 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:07:47.561 DEBUG:teuthology.orchestra.run.smithi117:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:07:47.563 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:07:47.563 DEBUG:teuthology.orchestra.run.smithi037:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:47.570 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:47.570+0000 7f4033642040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:07:47.670 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:47.670+0000 7f6850e56040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:07:47.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.689+0000 7fd8d8c31640 10 monclient: get_monmap_and_config 2024-03-25T18:07:47.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.689+0000 7fd8d8c31640 10 monclient: build_initial_monmap 2024-03-25T18:07:47.690 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.689+0000 7fd8d8c31640 10 monclient: monmap: 2024-03-25T18:07:47.690 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-03-25T18:07:47.690 INFO:teuthology.orchestra.run.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.690 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-25T18:07:47.690277+0000 2024-03-25T18:07:47.690 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-25T18:07:47.690277+0000 2024-03-25T18:07:47.690 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:47.690 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:47.690 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:47.691 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:47.691 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:47.691 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:47.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 1 Processor -- start 2024-03-25T18:07:47.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 1 -- start start 2024-03-25T18:07:47.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 10 monclient: init 2024-03-25T18:07:47.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:47.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:07:47.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 10 monclient(hunting): picked mon.noname-b con 0x7fd8d40b4700 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:47.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 10 monclient(hunting): picked mon.noname-c con 0x7fd8d4142a20 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:47.692 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 10 monclient(hunting): picked mon.noname-a con 0x7fd8d40a96b0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:47.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d40a9d60 con 0x7fd8d40a96b0 2024-03-25T18:07:47.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d413ff10 con 0x7fd8d4142a20 2024-03-25T18:07:47.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d40aa1f0 con 0x7fd8d40b4700 2024-03-25T18:07:47.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:47.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.690+0000 7fd8d8c31640 10 monclient(hunting): authenticate will time out at 6822.943189s 2024-03-25T18:07:47.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.691+0000 7fd8d2d9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd8d40a96b0 0x7fd8d413f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36880/0 (socket says 172.21.15.37:36880) 2024-03-25T18:07:47.693 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.691+0000 7fd8d2d9d640 1 -- 172.21.15.37:0/2515196666 learned_addr learned my addr 172.21.15.37:0/2515196666 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:07:47.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.694+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3544018650 0 0) 0x7fd8d40a9d60 con 0x7fd8d40a96b0 2024-03-25T18:07:47.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.694+0000 7fd8d159a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.694+0000 7fd8d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.694+0000 7fd8d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.694+0000 7fd8d159a640 10 monclient(hunting): my global_id is 4161 2024-03-25T18:07:47.695 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.694+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8bc003520 con 0x7fd8d40a96b0 2024-03-25T18:07:47.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.695+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2601658862 0 0) 0x7fd8d413ff10 con 0x7fd8d4142a20 2024-03-25T18:07:47.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.695+0000 7fd8d159a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.695+0000 7fd8d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.695+0000 7fd8d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.695+0000 7fd8d159a640 10 monclient(hunting): my global_id is 4162 2024-03-25T18:07:47.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.695+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d40a9d60 con 0x7fd8d4142a20 2024-03-25T18:07:47.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.696+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559764081 0 0) 0x7fd8d40aa1f0 con 0x7fd8d40b4700 2024-03-25T18:07:47.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.696+0000 7fd8d159a640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.696+0000 7fd8d159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.696+0000 7fd8d159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.697 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.696+0000 7fd8d159a640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:07:47.698 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.696+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d413ff10 con 0x7fd8d40b4700 2024-03-25T18:07:47.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1233293680 0 0) 0x7fd8bc003520 con 0x7fd8d40a96b0 2024-03-25T18:07:47.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 >> v1:172.21.15.37:6790/0 conn(0x7fd8d40b4700 legacy=0x7fd8d4142310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 >> v1:172.21.15.117:6789/0 conn(0x7fd8d4142a20 legacy=0x7fd8d413f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 10 monclient: _finish_hunting 0 2024-03-25T18:07:47.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 1 monclient: found mon.noname-a 2024-03-25T18:07:47.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:47.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:07:47.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d40b4ae0 con 0x7fd8d40a96b0 2024-03-25T18:07:47.700 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 10 monclient: _finish_auth 0 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 10 monclient: _check_auth_tickets 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd8cc0019d0 con 0x7fd8d40a96b0 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.697+0000 7fd8d159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.698+0000 7fd8d159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.698+0000 7fd8d159a640 10 monclient: dump: 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:47.701 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:47.702 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:47.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.698+0000 7fd8d8c31640 5 monclient: authenticate success, global_id 4161 2024-03-25T18:07:47.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.698+0000 7fd8d8c31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:47.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.699+0000 7fd8d159a640 1 -- 172.21.15.37:0/2515196666 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8cc001c20 con 0x7fd8d40a96b0 2024-03-25T18:07:47.702 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.699+0000 7fd8d159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:47.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.699+0000 7fd8d8c31640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:47.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.699+0000 7fd8d8c31640 4 set_mon_vals no callback set 2024-03-25T18:07:47.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.700+0000 7fd8d8c31640 10 monclient: shutdown 2024-03-25T18:07:47.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.700+0000 7fd8d8c31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:47.705 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.700+0000 7fd8d8c31640 1 -- 172.21.15.37:0/2515196666 >> v1:172.21.15.37:6789/0 conn(0x7fd8d40a96b0 legacy=0x7fd8d413f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.701+0000 7fd8d8c31640 1 -- 172.21.15.37:0/2515196666 shutdown_connections 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.701+0000 7fd8d8c31640 1 -- 172.21.15.37:0/2515196666 >> 172.21.15.37:0/2515196666 conn(0x7fd8d40668e0 msgr2=0x7fd8d4066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.701+0000 7fd8d8c31640 1 -- 172.21.15.37:0/2515196666 shutdown_connections 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.701+0000 7fd8d8c31640 1 -- 172.21.15.37:0/2515196666 wait complete. 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.702+0000 7fd8d8c31640 10 monclient: build_initial_monmap 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.702+0000 7fd8d8c31640 10 monclient: monmap: 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 0 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 0.000000 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:created 0.000000 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 1 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:47.706 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.702+0000 7fd8d8c31640 1 librados: starting msgr at 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.702+0000 7fd8d8c31640 1 librados: starting objecter 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 1 Processor -- start 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 1 -- start start 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 1 librados: setting wanted keys 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 1 librados: calling monclient init 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 10 monclient: init 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 10 monclient(hunting): picked mon.noname-a con 0x7fd8d40a96b0 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 10 monclient(hunting): picked mon.noname-b con 0x7fd8d40b4700 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 10 monclient(hunting): picked mon.noname-c con 0x7fd8d4142a20 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d40ac8c0 con 0x7fd8d40a96b0 2024-03-25T18:07:47.707 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d40710f0 con 0x7fd8d40b4700 2024-03-25T18:07:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd8d40ac090 con 0x7fd8d4142a20 2024-03-25T18:07:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.703+0000 7fd8d8c31640 10 monclient(hunting): authenticate will time out at 6822.956134s 2024-03-25T18:07:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.704+0000 7fd8d259c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd8d40a96b0 0x7fd8d41dc360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36890/0 (socket says 172.21.15.37:36890) 2024-03-25T18:07:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.704+0000 7fd8d259c640 1 -- 172.21.15.37:0/1997132765 learned_addr learned my addr 172.21.15.37:0/1997132765 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:07:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.705+0000 7fd8c2ffd640 10 client.?.objecter ms_handle_connect 0x7fd8d40a96b0 2024-03-25T18:07:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.705+0000 7fd8c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:47.708 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.706+0000 7fd8c2ffd640 10 client.?.objecter ms_handle_connect 0x7fd8d40b4700 2024-03-25T18:07:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.706+0000 7fd8c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.706+0000 7fd8c2ffd640 10 client.?.objecter ms_handle_connect 0x7fd8d4142a20 2024-03-25T18:07:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.706+0000 7fd8c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.707+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1785306798 0 0) 0x7fd8d40ac8c0 con 0x7fd8d40a96b0 2024-03-25T18:07:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.707+0000 7fd8c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.707+0000 7fd8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.707+0000 7fd8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.707+0000 7fd8c2ffd640 10 monclient(hunting): my global_id is 4164 2024-03-25T18:07:47.709 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.707+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8b4004eb0 con 0x7fd8d40a96b0 2024-03-25T18:07:47.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.708+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244360902 0 0) 0x7fd8d40710f0 con 0x7fd8d40b4700 2024-03-25T18:07:47.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.708+0000 7fd8c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.709+0000 7fd8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.709+0000 7fd8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.709+0000 7fd8c2ffd640 10 monclient(hunting): my global_id is 4165 2024-03-25T18:07:47.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.709+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d40ac8c0 con 0x7fd8d40b4700 2024-03-25T18:07:47.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.709+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393983454 0 0) 0x7fd8d40ac090 con 0x7fd8d4142a20 2024-03-25T18:07:47.710 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.709+0000 7fd8c2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:47.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.709+0000 7fd8c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:47.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.709+0000 7fd8c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:47.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.709+0000 7fd8c2ffd640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:07:47.711 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.709+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd8d40710f0 con 0x7fd8d4142a20 2024-03-25T18:07:47.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.714+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1253400233 0 0) 0x7fd8b4004eb0 con 0x7fd8d40a96b0 2024-03-25T18:07:47.714 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.714+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8d40ac090 con 0x7fd8d40a96b0 2024-03-25T18:07:47.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.715+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd8c4001ad0 con 0x7fd8d40a96b0 2024-03-25T18:07:47.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.715+0000 7fd8c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:47.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.715+0000 7fd8c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:47.716 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.715+0000 7fd8c2ffd640 10 monclient(hunting): dump: 2024-03-25T18:07:47.716 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-25T18:07:47.716 INFO:teuthology.orchestra.run.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.716 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.716 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.716 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:47.717 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:47.717 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:47.717 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:47.717 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:47.717 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:47.717 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.715+0000 7fd8c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:47.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.716+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 614425680 0 0) 0x7fd8d40710f0 con 0x7fd8d4142a20 2024-03-25T18:07:47.718 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.716+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8b4004eb0 con 0x7fd8d4142a20 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.717+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2082275821 0 0) 0x7fd8d40ac8c0 con 0x7fd8d40b4700 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.717+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd8d40710f0 con 0x7fd8d40b4700 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.717+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd8cc0019d0 con 0x7fd8d4142a20 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.717+0000 7fd8c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.717+0000 7fd8c2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.717+0000 7fd8c2ffd640 10 monclient(hunting): dump: 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:47.719 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.717+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd8c80017f0 con 0x7fd8d40b4700 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.717+0000 7fd8c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.717+0000 7fd8c2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.717+0000 7fd8c2ffd640 10 monclient(hunting): dump: 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:47.720 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:47.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2530572877 0 0) 0x7fd8d40ac090 con 0x7fd8d40a96b0 2024-03-25T18:07:47.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 >> v1:172.21.15.37:6790/0 conn(0x7fd8d4142a20 legacy=0x7fd8d412ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 >> v1:172.21.15.117:6789/0 conn(0x7fd8d40b4700 legacy=0x7fd8d40ab980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:47.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8c2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:47.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8c2ffd640 1 monclient: found mon.a 2024-03-25T18:07:47.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:47.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8c2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:47.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d40acb30 con 0x7fd8d40a96b0 2024-03-25T18:07:47.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8c2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:07:47.724 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8c2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8d8c31640 5 monclient: authenticate success, global_id 4164 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8d8c31640 10 monclient: _renew_subs 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8d8c31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.721+0000 7fd8d8c31640 1 -- 172.21.15.37:0/1997132765 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8d40ace10 con 0x7fd8d40a96b0 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.722+0000 7fd8d8c31640 10 client.4164.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.722+0000 7fd8d8c31640 10 monclient: _renew_subs 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.722+0000 7fd8d8c31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.722+0000 7fd8d8c31640 1 -- 172.21.15.37:0/1997132765 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8d4130610 con 0x7fd8d40a96b0 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.722+0000 7fd8d8c31640 1 librados: init done 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.723+0000 7fd8d8c31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.723+0000 7fd8d8c31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.723+0000 7fd8d8c31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:47.725 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.723+0000 7fd8d8c31640 1 -- 172.21.15.37:0/1997132765 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8d404f730 con 0x7fd8d40a96b0 2024-03-25T18:07:47.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.725+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd8c4001c30 con 0x7fd8d40a96b0 2024-03-25T18:07:47.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.725+0000 7fd8c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:47.726 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.725+0000 7fd8c3fff640 4 set_mon_vals no callback set 2024-03-25T18:07:47.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.726+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd8c4003ae0 con 0x7fd8d40a96b0 2024-03-25T18:07:47.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.726+0000 7fd8c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:47.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.726+0000 7fd8c2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:47.727 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.726+0000 7fd8c2ffd640 10 monclient: dump: 2024-03-25T18:07:47.727 INFO:teuthology.orchestra.run.smithi037.stderr:epoch 1 2024-03-25T18:07:47.727 INFO:teuthology.orchestra.run.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:47.727 INFO:teuthology.orchestra.run.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.727 INFO:teuthology.orchestra.run.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:47.727 INFO:teuthology.orchestra.run.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:47.727 INFO:teuthology.orchestra.run.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:47.728 INFO:teuthology.orchestra.run.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:47.728 INFO:teuthology.orchestra.run.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:47.728 INFO:teuthology.orchestra.run.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:47.728 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:07:47.728 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.728+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd8c4004050 con 0x7fd8d40a96b0 2024-03-25T18:07:47.749 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:47.748+0000 7f4033642040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:07:47.750 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:47.750+0000 7f6850e56040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:07:47.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.914+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fd8c4005e90 con 0x7fd8d40a96b0 2024-03-25T18:07:47.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.914+0000 7fd8c2ffd640 10 client.4164.objecter ms_dispatch 0x7fd8d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:47.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.914+0000 7fd8c2ffd640 3 client.4164.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:47.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.915+0000 7fd8c2ffd640 3 client.4164.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:47.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.915+0000 7fd8c2ffd640 20 client.4164.objecter dump_active .. 0 homeless 2024-03-25T18:07:47.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.915+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fd8c4004360 con 0x7fd8d40a96b0 2024-03-25T18:07:47.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.915+0000 7fd8c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:47.916 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:47.915+0000 7fd8c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:47.969 INFO:tasks.ceph.mgr.y.smithi037.stderr:2024-03-25T18:07:47.968+0000 7f6850e56040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:07:48.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.040+0000 7fd8d8c31640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:48.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.040+0000 7fd8d8c31640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:48.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.040+0000 7fd8d8c31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:48.041 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.040+0000 7fd8d8c31640 1 -- 172.21.15.37:0/1997132765 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd8d4071280 con 0x7fd8d40a96b0 2024-03-25T18:07:48.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.088+0000 7fd8c2ffd640 1 -- 172.21.15.37:0/1997132765 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fd8c40057c0 con 0x7fd8d40a96b0 2024-03-25T18:07:48.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.088+0000 7fd8c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:48.089 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.088+0000 7fd8c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:48.089 INFO:teuthology.orchestra.run.smithi037.stdout: 2024-03-25T18:07:48.089 INFO:teuthology.orchestra.run.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:48.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.089+0000 7fd8d8c31640 10 librados: watch_flush enter 2024-03-25T18:07:48.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.089+0000 7fd8d8c31640 10 librados: watch_flush exit 2024-03-25T18:07:48.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.089+0000 7fd8d8c31640 20 client.4164.objecter shutdown clearing up homeless session... 2024-03-25T18:07:48.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.089+0000 7fd8d8c31640 10 client.4164.objecter successfully canceled tick 2024-03-25T18:07:48.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.090+0000 7fd8d8c31640 10 monclient: shutdown 2024-03-25T18:07:48.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.090+0000 7fd8d8c31640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:48.091 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.090+0000 7fd8d8c31640 1 -- 172.21.15.37:0/1997132765 >> v1:172.21.15.37:6789/0 conn(0x7fd8d40a96b0 legacy=0x7fd8d41dc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:48.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.090+0000 7fd8d8c31640 1 -- 172.21.15.37:0/1997132765 shutdown_connections 2024-03-25T18:07:48.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.090+0000 7fd8d8c31640 1 -- 172.21.15.37:0/1997132765 >> 172.21.15.37:0/1997132765 conn(0x7fd8d40668e0 msgr2=0x7fd8d4066cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:48.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.090+0000 7fd8d8c31640 1 -- 172.21.15.37:0/1997132765 shutdown_connections 2024-03-25T18:07:48.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.090+0000 7fd8d8c31640 1 -- 172.21.15.37:0/1997132765 wait complete. 2024-03-25T18:07:48.092 INFO:teuthology.orchestra.run.smithi037.stderr:2024-03-25T18:07:48.090+0000 7fd8d8c31640 1 librados: shutdown 2024-03-25T18:07:48.107 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:07:48.107 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:07:48.912 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:49.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.056+0000 7f08681c7640 10 monclient: get_monmap_and_config 2024-03-25T18:07:49.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.056+0000 7f08681c7640 10 monclient: build_initial_monmap 2024-03-25T18:07:49.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.056+0000 7f08681c7640 10 monclient: monmap: 2024-03-25T18:07:49.057 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:07:49.057 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:49.057 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:07:49.057345+0000 2024-03-25T18:07:49.058 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:07:49.057345+0000 2024-03-25T18:07:49.058 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:49.058 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:49.058 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:49.058 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:49.058 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:49.058 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.057+0000 7f08681c7640 1 Processor -- start 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.057+0000 7f08681c7640 1 -- start start 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.057+0000 7f08681c7640 10 monclient: init 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.058+0000 7f08681c7640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.058+0000 7f08681c7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.058+0000 7f08681c7640 10 monclient(hunting): picked mon.noname-a con 0x7f08600b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.058+0000 7f08681c7640 10 monclient(hunting): picked mon.noname-b con 0x7f08600b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.058+0000 7f08681c7640 10 monclient(hunting): picked mon.noname-c con 0x7f086014be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.058+0000 7f08681c7640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08600af940 con 0x7f08600b4d60 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.059+0000 7f08681c7640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08600b0a70 con 0x7f086014be90 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.059+0000 7f08681c7640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08600b0d50 con 0x7f08600b1730 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.059+0000 7f08681c7640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.059+0000 7f08681c7640 10 monclient(hunting): authenticate will time out at 6824.311617s 2024-03-25T18:07:49.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.059+0000 7f0865762640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f08600b1730 0x7f08600b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:50786/0 (socket says 172.21.15.37:50786) 2024-03-25T18:07:49.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.059+0000 7f0865762640 1 -- 172.21.15.37:0/167673870 learned_addr learned my addr 172.21.15.37:0/167673870 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:07:49.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.087+0000 7f0864f61640 1 -- 172.21.15.37:0/167673870 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157431005 0 0) 0x7f08600af940 con 0x7f08600b4d60 2024-03-25T18:07:49.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.087+0000 7f0864f61640 10 monclient(hunting): _negotiate 2024-03-25T18:07:49.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.087+0000 7f0864f61640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:49.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.087+0000 7f0864f61640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:49.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.087+0000 7f0864f61640 10 monclient(hunting): my global_id is 4170 2024-03-25T18:07:49.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.087+0000 7f0864f61640 1 -- 172.21.15.37:0/167673870 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f084c003520 con 0x7f08600b4d60 2024-03-25T18:07:49.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.098+0000 7f0864f61640 1 -- 172.21.15.37:0/167673870 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1654937517 0 0) 0x7f084c003520 con 0x7f08600b4d60 2024-03-25T18:07:49.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.098+0000 7f0864f61640 1 -- 172.21.15.37:0/167673870 >> v1:172.21.15.37:6790/0 conn(0x7f08600b1730 legacy=0x7f08600b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.098+0000 7f0864f61640 1 -- 172.21.15.37:0/167673870 >> v1:172.21.15.117:6789/0 conn(0x7f086014be90 legacy=0x7f086014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.098+0000 7f0864f61640 10 monclient: _finish_hunting 0 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.098+0000 7f0864f61640 1 monclient: found mon.noname-a 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.098+0000 7f0864f61640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.098+0000 7f0864f61640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.098+0000 7f0864f61640 1 -- 172.21.15.37:0/167673870 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08600b1030 con 0x7f08600b4d60 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.098+0000 7f0864f61640 10 monclient: _finish_auth 0 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.098+0000 7f0864f61640 10 monclient: _check_auth_tickets 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.098+0000 7f0864f61640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.099+0000 7f0864f61640 1 -- 172.21.15.37:0/167673870 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f085c001690 con 0x7f08600b4d60 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.099+0000 7f08681c7640 5 monclient: authenticate success, global_id 4170 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.099+0000 7f08681c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.099+0000 7f0864f61640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.099+0000 7f0864f61640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.099+0000 7f0864f61640 10 monclient: dump: 2024-03-25T18:07:49.100 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:07:49.101 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:49.101 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:49.101 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:49.101 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:49.101 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:49.101 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:49.101 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:49.101 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:49.101 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:49.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.099+0000 7f08681c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.215+0000 7f0864f61640 1 -- 172.21.15.37:0/167673870 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f085c001920 con 0x7f08600b4d60 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.215+0000 7f0864f61640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.215+0000 7f0864f61640 1 -- 172.21.15.37:0/167673870 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f085c001cb0 con 0x7f08600b4d60 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.215+0000 7f08681c7640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.215+0000 7f08681c7640 4 set_mon_vals no callback set 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.215+0000 7f0864f61640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.215+0000 7f0864f61640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.215+0000 7f0864f61640 10 monclient: dump: 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:49.216 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:49.217 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:49.217 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:49.217 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:49.217 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:49.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.216+0000 7f08681c7640 10 monclient: shutdown 2024-03-25T18:07:49.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.216+0000 7f08681c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:49.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.216+0000 7f08681c7640 1 -- 172.21.15.37:0/167673870 >> v1:172.21.15.37:6789/0 conn(0x7f08600b4d60 legacy=0x7f086013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:49.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.216+0000 7f08681c7640 1 -- 172.21.15.37:0/167673870 shutdown_connections 2024-03-25T18:07:49.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.216+0000 7f08681c7640 1 -- 172.21.15.37:0/167673870 >> 172.21.15.37:0/167673870 conn(0x7f08600617a0 msgr2=0x7f0860061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:49.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.216+0000 7f08681c7640 1 -- 172.21.15.37:0/167673870 shutdown_connections 2024-03-25T18:07:49.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.216+0000 7f08681c7640 1 -- 172.21.15.37:0/167673870 wait complete. 2024-03-25T18:07:49.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.217+0000 7f08681c7640 10 monclient: build_initial_monmap 2024-03-25T18:07:49.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.217+0000 7f08681c7640 10 monclient: monmap: 2024-03-25T18:07:49.219 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:07:49.219 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:49.219 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:07:49.219 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:07:49.219 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:49.219 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:07:49.219 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:49.219 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:49.220 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:49.220 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:49.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.218+0000 7f08681c7640 1 librados: starting msgr at 2024-03-25T18:07:49.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.218+0000 7f08681c7640 1 librados: starting objecter 2024-03-25T18:07:49.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.218+0000 7f08681c7640 1 Processor -- start 2024-03-25T18:07:49.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.218+0000 7f08681c7640 1 -- start start 2024-03-25T18:07:49.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.218+0000 7f08681c7640 1 librados: setting wanted keys 2024-03-25T18:07:49.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.218+0000 7f08681c7640 1 librados: calling monclient init 2024-03-25T18:07:49.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.218+0000 7f08681c7640 10 monclient: init 2024-03-25T18:07:49.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.219+0000 7f08681c7640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.219+0000 7f08681c7640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.219+0000 7f08681c7640 10 monclient(hunting): picked mon.noname-a con 0x7f08600b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.220+0000 7f0865762640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f08600b4d60 0x7f0860137000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:50796/0 (socket says 172.21.15.37:50796) 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.220+0000 7f0865762640 1 -- 172.21.15.37:0/3741279298 learned_addr learned my addr 172.21.15.37:0/3741279298 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.220+0000 7f08681c7640 10 monclient(hunting): picked mon.noname-c con 0x7f08600b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.220+0000 7f08681c7640 10 monclient(hunting): picked mon.noname-b con 0x7f086014be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.220+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086013b350 con 0x7f08600b1730 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.220+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086013b7e0 con 0x7f086014be90 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.220+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f086013a1e0 con 0x7f08600b4d60 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.220+0000 7f08681c7640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.220+0000 7f08681c7640 10 monclient(hunting): authenticate will time out at 6824.472745s 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.220+0000 7f08567fc640 10 client.?.objecter ms_handle_connect 0x7f08600b4d60 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.220+0000 7f08567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:49.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.221+0000 7f08567fc640 10 client.?.objecter ms_handle_connect 0x7f086014be90 2024-03-25T18:07:49.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.221+0000 7f08567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:49.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.252+0000 7f08567fc640 10 client.?.objecter ms_handle_connect 0x7f08600b1730 2024-03-25T18:07:49.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.252+0000 7f08567fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:49.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.569+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538331910 0 0) 0x7f086013b350 con 0x7f08600b1730 2024-03-25T18:07:49.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.569+0000 7f08567fc640 10 monclient(hunting): _negotiate 2024-03-25T18:07:49.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.569+0000 7f08567fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:49.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.569+0000 7f08567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:49.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.569+0000 7f08567fc640 10 monclient(hunting): my global_id is 4173 2024-03-25T18:07:49.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.569+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0844004eb0 con 0x7f08600b1730 2024-03-25T18:07:49.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.579+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4252396012 0 0) 0x7f0844004eb0 con 0x7f08600b1730 2024-03-25T18:07:49.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.579+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f086013b350 con 0x7f08600b1730 2024-03-25T18:07:49.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.579+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f085c001610 con 0x7f08600b1730 2024-03-25T18:07:49.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.579+0000 7f08567fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:49.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.579+0000 7f08567fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:07:49.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.579+0000 7f08567fc640 10 monclient(hunting): dump: 2024-03-25T18:07:49.580 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:07:49.580 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:49.581 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:49.581 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:49.581 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:49.581 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:49.581 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:49.581 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:49.581 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:49.581 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:49.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.579+0000 7f08567fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:49.590 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 589278725 0 0) 0x7f086013b350 con 0x7f08600b1730 2024-03-25T18:07:49.590 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 >> v1:172.21.15.37:6790/0 conn(0x7f08600b4d60 legacy=0x7f0860137000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:49.590 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 >> v1:172.21.15.117:6789/0 conn(0x7f086014be90 legacy=0x7f08601d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:49.590 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08567fc640 10 monclient: _finish_hunting 0 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08567fc640 1 monclient: found mon.a 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08567fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0860137710 con 0x7f08600b1730 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08567fc640 10 monclient: _finish_auth 0 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08567fc640 10 monclient: _check_auth_tickets 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08681c7640 5 monclient: authenticate success, global_id 4173 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08681c7640 10 monclient: _renew_subs 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08681c7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08601426c0 con 0x7f08600b1730 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08681c7640 10 client.4173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08681c7640 10 monclient: _renew_subs 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08681c7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f08601d7ca0 con 0x7f08600b1730 2024-03-25T18:07:49.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.589+0000 7f08681c7640 1 librados: init done 2024-03-25T18:07:49.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.591+0000 7f08681c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:49.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.591+0000 7f08681c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:49.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.591+0000 7f08681c7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:49.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.591+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0860137960 con 0x7f08600b1730 2024-03-25T18:07:49.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.645+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f085c003960 con 0x7f08600b1730 2024-03-25T18:07:49.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.645+0000 7f08567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:49.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.645+0000 7f0857fff640 4 set_mon_vals no callback set 2024-03-25T18:07:49.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.647+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f085c001250 con 0x7f08600b1730 2024-03-25T18:07:49.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.647+0000 7f08567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:49.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.647+0000 7f08567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:49.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.647+0000 7f08567fc640 10 monclient: dump: 2024-03-25T18:07:49.648 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:07:49.648 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:49.648 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:49.649 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:49.649 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:49.649 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:49.649 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:49.649 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:49.649 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:49.649 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:49.688 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:49.687+0000 7f3fd4351640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:07:49.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.700+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 689902368 0 0) 0x7f085c019a70 con 0x7f08600b1730 2024-03-25T18:07:49.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.701+0000 7f0865762640 10 monclient: get_auth_request con 0x7f084406dbc0 auth_method 0 2024-03-25T18:07:49.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:49.701+0000 7f08567fc640 10 client.4173.objecter ms_handle_connect 0x7f084406dbc0 2024-03-25T18:07:49.706 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:07:49.706+0000 7f3fd4351640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:07:50.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.122+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f085c053710 con 0x7f08600b1730 2024-03-25T18:07:50.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.122+0000 7f08567fc640 10 client.4173.objecter ms_dispatch 0x7f0860000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:50.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.122+0000 7f08567fc640 3 client.4173.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:50.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.122+0000 7f08567fc640 3 client.4173.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:50.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.122+0000 7f08567fc640 20 client.4173.objecter dump_active .. 0 homeless 2024-03-25T18:07:50.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.122+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f085c04f220 con 0x7f08600b1730 2024-03-25T18:07:50.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.122+0000 7f08567fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:50.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.122+0000 7f08567fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:50.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.475+0000 7f08681c7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:50.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.475+0000 7f08681c7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:50.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.475+0000 7f08681c7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:07:50.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.475+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0860139f90 con 0x7f08600b1730 2024-03-25T18:07:50.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.585+0000 7f08567fc640 1 -- 172.21.15.37:0/3741279298 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f085c051150 con 0x7f08600b1730 2024-03-25T18:07:50.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.585+0000 7f08567fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:50.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.585+0000 7f08567fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:50.586 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:07:50.586 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:50.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.587+0000 7f08681c7640 10 librados: watch_flush enter 2024-03-25T18:07:50.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.587+0000 7f08681c7640 10 librados: watch_flush exit 2024-03-25T18:07:50.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.587+0000 7f08681c7640 20 client.4173.objecter shutdown clearing up homeless session... 2024-03-25T18:07:50.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.587+0000 7f08681c7640 10 client.4173.objecter successfully canceled tick 2024-03-25T18:07:50.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.588+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 >> v1:172.21.15.117:6800/3703993921 conn(0x7f084406dbc0 legacy=0x7f0844070080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:50.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.588+0000 7f08681c7640 10 monclient: shutdown 2024-03-25T18:07:50.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.588+0000 7f08681c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:50.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.588+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 >> v1:172.21.15.37:6789/0 conn(0x7f08600b1730 legacy=0x7f086013a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:50.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.588+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 shutdown_connections 2024-03-25T18:07:50.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.588+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 >> 172.21.15.37:0/3741279298 conn(0x7f08600617a0 msgr2=0x7f0860145110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:50.590 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.588+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 shutdown_connections 2024-03-25T18:07:50.590 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.588+0000 7f08681c7640 1 -- 172.21.15.37:0/3741279298 wait complete. 2024-03-25T18:07:50.590 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:50.588+0000 7f08681c7640 1 librados: shutdown 2024-03-25T18:07:50.604 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:54.425 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:06.800 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:54.486 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:06.860 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:54.529 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:06.884 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:54.553 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:06.933 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:54.590 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:06.945 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:54.616 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:06.995 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:54.616 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:06.972 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:54.676 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:07.031 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:54.925 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:07.288 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:54.990 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:07.352 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.025 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:07.399 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.037 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-03-25T18:07:55.031+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:07:55.086 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:07.460 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.162 INFO:tasks.ceph.osd.4.smithi117.stderr:2024-03-25T18:07:55.156+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:07:55.166 INFO:tasks.ceph.osd.5.smithi117.stderr:2024-03-25T18:07:55.160+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:07:55.232 INFO:tasks.ceph.osd.7.smithi117.stderr:2024-03-25T18:07:55.225+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:07:55.353 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:07.731 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.413 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:07.792 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.513 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:07.887 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.536 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-03-25T18:07:55.529+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:07:55.629 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:07.984 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.629 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:08.009 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.630 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:07.996 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.661 INFO:tasks.ceph.osd.3.smithi037.stderr:2024-03-25T18:07:55.653+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:07:55.692 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:08.057 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.706 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:08.062 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.875 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:08.250 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.968 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-03-25T18:07:55.961+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:07:55.996 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:08.376 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:55.999 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:08.355 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.001 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:08.364 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.081 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:08.436 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.134 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:08.509 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.149 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:08.504 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.234 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-03-25T18:07:56.227+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:07:56.238 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:08.613 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.308 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:08.688 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.343 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:08.698 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.370 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:08.733 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.413 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:08.768 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.455 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:08.834 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.523 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:08.897 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.528 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:08.903 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.576 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:08.956 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.609 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:08.964 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.677 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:09.032 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.686 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:09.049 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.696 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:09.061 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.815 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:09.190 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.821 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:09.199 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.840 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:09.214 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.843 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:09.223 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.874 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:09.229 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.946 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:09.301 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:56.956 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:09.319 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.062 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:09.427 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.109 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:09.483 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.113 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:09.487 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.116 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:09.496 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.145 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:09.500 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.146 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:09.525 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.214 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:09.569 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.224 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:09.587 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.379 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:09.745 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.380 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:09.760 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.386 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:09.760 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.407 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:09.781 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.409 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:07:57.416 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:09.794 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.427 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:09.782 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.478 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:09.833 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.498+0000 7f15327ab640 10 monclient: get_monmap_and_config 2024-03-25T18:07:57.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.498+0000 7f15327ab640 10 monclient: build_initial_monmap 2024-03-25T18:07:57.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.498+0000 7f15327ab640 10 monclient: monmap: 2024-03-25T18:07:57.499 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:07:57.499 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:57.499 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:07:57.499444+0000 2024-03-25T18:07:57.499 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:07:57.499444+0000 2024-03-25T18:07:57.499 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:57.499 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:57.500 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:57.500 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:07:57.500 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:07:57.500 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:57.500 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:09.862 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.501 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 1 Processor -- start 2024-03-25T18:07:57.501 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 1 -- start start 2024-03-25T18:07:57.501 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 10 monclient: init 2024-03-25T18:07:57.501 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:57.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:07:57.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 10 monclient(hunting): picked mon.noname-c con 0x7f152c0b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:57.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 10 monclient(hunting): picked mon.noname-b con 0x7f152c0b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:57.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 10 monclient(hunting): picked mon.noname-a con 0x7f152c14be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:57.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c0af940 con 0x7f152c14be90 2024-03-25T18:07:57.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c0b0a70 con 0x7f152c0b4d60 2024-03-25T18:07:57.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c0b0d50 con 0x7f152c0b1730 2024-03-25T18:07:57.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:57.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.499+0000 7f15327ab640 10 monclient(hunting): authenticate will time out at 6832.752191s 2024-03-25T18:07:57.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.500+0000 7f152b7fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f152c0b1730 0x7f152c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:50830/0 (socket says 172.21.15.37:50830) 2024-03-25T18:07:57.502 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.500+0000 7f152b7fe640 1 -- 172.21.15.37:0/4201299128 learned_addr learned my addr 172.21.15.37:0/4201299128 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.502+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3882806974 0 0) 0x7f152c0b0d50 con 0x7f152c0b1730 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.502+0000 7f152affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.502+0000 7f152affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.502+0000 7f152affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.502+0000 7f152affd640 10 monclient(hunting): my global_id is 4202 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.502+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1514003520 con 0x7f152c0b1730 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3935077348 0 0) 0x7f152c0b0a70 con 0x7f152c0b4d60 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f152c0b0d50 con 0x7f152c0b4d60 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 151915136 0 0) 0x7f152c0af940 con 0x7f152c14be90 2024-03-25T18:07:57.504 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 10 monclient(hunting): _negotiate 2024-03-25T18:07:57.505 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:57.505 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:57.505 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 10 monclient(hunting): my global_id is 4200 2024-03-25T18:07:57.505 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.503+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f152c0b0a70 con 0x7f152c14be90 2024-03-25T18:07:57.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2662593071 0 0) 0x7f152c0b0d50 con 0x7f152c0b4d60 2024-03-25T18:07:57.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 >> v1:172.21.15.37:6790/0 conn(0x7f152c0b1730 legacy=0x7f152c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:57.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 >> v1:172.21.15.37:6789/0 conn(0x7f152c14be90 legacy=0x7f152c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:57.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 10 monclient: _finish_hunting 0 2024-03-25T18:07:57.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 1 monclient: found mon.noname-c 2024-03-25T18:07:57.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:57.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:07:57.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c0b1030 con 0x7f152c0b4d60 2024-03-25T18:07:57.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 10 monclient: _finish_auth 0 2024-03-25T18:07:57.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 10 monclient: _check_auth_tickets 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f15200015a0 con 0x7f152c0b4d60 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f15327ab640 5 monclient: authenticate success, global_id 4201 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f15327ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f152affd640 10 monclient: dump: 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:57.508 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:57.509 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:57.509 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:57.509 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:57.509 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:57.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.506+0000 7f15327ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:07:57.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.507+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1520001700 con 0x7f152c0b4d60 2024-03-25T18:07:57.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.507+0000 7f152affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:57.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.507+0000 7f152affd640 1 -- 172.21.15.37:0/4201299128 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1520001b50 con 0x7f152c0b4d60 2024-03-25T18:07:57.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.507+0000 7f152affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:57.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.507+0000 7f152affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:57.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.507+0000 7f152affd640 10 monclient: dump: 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.507+0000 7f15327ab640 10 monclient: get_monmap_and_config success 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.507+0000 7f15327ab640 4 set_mon_vals no callback set 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.508+0000 7f15327ab640 10 monclient: shutdown 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.508+0000 7f15327ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.508+0000 7f15327ab640 1 -- 172.21.15.37:0/4201299128 >> v1:172.21.15.117:6789/0 conn(0x7f152c0b4d60 legacy=0x7f152c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:57.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.508+0000 7f15327ab640 1 -- 172.21.15.37:0/4201299128 shutdown_connections 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.508+0000 7f15327ab640 1 -- 172.21.15.37:0/4201299128 >> 172.21.15.37:0/4201299128 conn(0x7f152c0617a0 msgr2=0x7f152c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.508+0000 7f15327ab640 1 -- 172.21.15.37:0/4201299128 shutdown_connections 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.508+0000 7f15327ab640 1 -- 172.21.15.37:0/4201299128 wait complete. 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.509+0000 7f15327ab640 10 monclient: build_initial_monmap 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.509+0000 7f15327ab640 10 monclient: monmap: 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:07:57.513 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.509+0000 7f15327ab640 1 librados: starting msgr at 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.509+0000 7f15327ab640 1 librados: starting objecter 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.509+0000 7f15327ab640 1 Processor -- start 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.509+0000 7f15327ab640 1 -- start start 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.509+0000 7f15327ab640 1 librados: setting wanted keys 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.509+0000 7f15327ab640 1 librados: calling monclient init 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.509+0000 7f15327ab640 10 monclient: init 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f15327ab640 10 monclient: _reopen_session rank -1 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f15327ab640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f15327ab640 10 monclient(hunting): picked mon.noname-c con 0x7f152c0b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f15327ab640 10 monclient(hunting): picked mon.noname-a con 0x7f152c0b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f15327ab640 10 monclient(hunting): picked mon.noname-b con 0x7f152c14be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f15327ab640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c141ce0 con 0x7f152c0b4d60 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f15327ab640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c141fc0 con 0x7f152c14be90 2024-03-25T18:07:57.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f15327ab640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f152c13bc70 con 0x7f152c0b1730 2024-03-25T18:07:57.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f15327ab640 10 monclient(hunting): _renew_subs 2024-03-25T18:07:57.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f15327ab640 10 monclient(hunting): authenticate will time out at 6832.762585s 2024-03-25T18:07:57.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f152bfff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f152c0b1730 0x7f152c1db200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:50836/0 (socket says 172.21.15.37:50836) 2024-03-25T18:07:57.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.510+0000 7f152bfff640 1 -- 172.21.15.37:0/2636531935 learned_addr learned my addr 172.21.15.37:0/2636531935 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:07:57.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.511+0000 7f1528ff9640 10 client.?.objecter ms_handle_connect 0x7f152c0b1730 2024-03-25T18:07:57.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.511+0000 7f1528ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:57.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.511+0000 7f1528ff9640 10 client.?.objecter ms_handle_connect 0x7f152c0b4d60 2024-03-25T18:07:57.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.511+0000 7f1528ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:57.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.511+0000 7f1528ff9640 10 client.?.objecter ms_handle_connect 0x7f152c14be90 2024-03-25T18:07:57.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.511+0000 7f1528ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:07:57.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 493391059 0 0) 0x7f152c13bc70 con 0x7f152c0b1730 2024-03-25T18:07:57.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:57.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:57.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:57.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): my global_id is 4205 2024-03-25T18:07:57.516 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1518004eb0 con 0x7f152c0b1730 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2212906648 0 0) 0x7f152c141fc0 con 0x7f152c14be90 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f152c13bc70 con 0x7f152c14be90 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 786475580 0 0) 0x7f152c141ce0 con 0x7f152c0b4d60 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:07:57.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.513+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f152c141fc0 con 0x7f152c0b4d60 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4090771781 0 0) 0x7f152c13bc70 con 0x7f152c14be90 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f152c141ce0 con 0x7f152c14be90 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1524001b50 con 0x7f152c14be90 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 10 monclient(hunting): dump: 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:57.519 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 640249636 0 0) 0x7f1518004eb0 con 0x7f152c0b1730 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f152c13bc70 con 0x7f152c0b1730 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f15200015a0 con 0x7f152c0b1730 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.517+0000 7f1528ff9640 10 monclient(hunting): dump: 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:57.520 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.518+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3314891249 0 0) 0x7f152c141fc0 con 0x7f152c0b4d60 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.518+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1518004eb0 con 0x7f152c0b4d60 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.518+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f151c001230 con 0x7f152c0b4d60 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.518+0000 7f1528ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.518+0000 7f1528ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.518+0000 7f1528ff9640 10 monclient(hunting): dump: 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:57.521 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:57.522 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:57.522 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:57.522 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:57.522 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:57.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3853532425 0 0) 0x7f152c141ce0 con 0x7f152c14be90 2024-03-25T18:07:57.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 >> v1:172.21.15.37:6790/0 conn(0x7f152c0b1730 legacy=0x7f152c1db200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:57.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 >> v1:172.21.15.37:6789/0 conn(0x7f152c0b4d60 legacy=0x7f152c1423e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:57.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f1528ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:07:57.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f1528ff9640 1 monclient: found mon.b 2024-03-25T18:07:57.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f1528ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:07:57.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f1528ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:57.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f152c140760 con 0x7f152c14be90 2024-03-25T18:07:57.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f1528ff9640 10 monclient: _finish_auth 0 2024-03-25T18:07:57.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f1528ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f1528ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f15327ab640 5 monclient: authenticate success, global_id 4204 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f15327ab640 10 monclient: _renew_subs 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f15327ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f15327ab640 1 -- 172.21.15.37:0/2636531935 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f152c140c80 con 0x7f152c14be90 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f15327ab640 10 client.4204.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f15327ab640 10 monclient: _renew_subs 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f15327ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f15327ab640 1 -- 172.21.15.37:0/2636531935 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f152c140380 con 0x7f152c14be90 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.520+0000 7f15327ab640 1 librados: init done 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.521+0000 7f15327ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.521+0000 7f15327ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.521+0000 7f15327ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:57.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.521+0000 7f15327ab640 1 -- 172.21.15.37:0/2636531935 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f152c0afbd0 con 0x7f152c14be90 2024-03-25T18:07:57.644 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:10.024 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.651 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:10.016 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.655 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:10.029 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.694 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:10.049 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.695 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:10.073 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.699 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:10.074 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.729+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1524001f60 con 0x7f152c14be90 2024-03-25T18:07:57.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.729+0000 7f1528ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:07:57.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.729+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1524003b30 con 0x7f152c14be90 2024-03-25T18:07:57.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.729+0000 7f1528ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:07:57.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.729+0000 7f1528ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:07:57.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.729+0000 7f1528ff9640 10 monclient: dump: 2024-03-25T18:07:57.730 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:07:57.730 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:07:57.730 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.730 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:07:57.730 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:07:57.731 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:07:57.731 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:07:57.731 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:07:57.731 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:07:57.731 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:07:57.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.729+0000 7f152a7fc640 4 set_mon_vals no callback set 2024-03-25T18:07:57.743 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:10.098 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.764 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:10.127 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.781+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f1524019b00 con 0x7f152c14be90 2024-03-25T18:07:57.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.782+0000 7f152bfff640 10 monclient: get_auth_request con 0x7f151806e160 auth_method 0 2024-03-25T18:07:57.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.782+0000 7f1528ff9640 10 client.4204.objecter ms_handle_connect 0x7f151806e160 2024-03-25T18:07:57.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.783+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f1524053680 con 0x7f152c14be90 2024-03-25T18:07:57.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.783+0000 7f1528ff9640 10 client.4204.objecter ms_dispatch 0x7f152c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:07:57.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.783+0000 7f1528ff9640 3 client.4204.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:07:57.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.783+0000 7f1528ff9640 3 client.4204.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:07:57.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.783+0000 7f1528ff9640 20 client.4204.objecter dump_active .. 0 homeless 2024-03-25T18:07:57.786 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.786+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f152404f190 con 0x7f152c14be90 2024-03-25T18:07:57.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.786+0000 7f1528ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:07:57.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.786+0000 7f1528ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:07:57.915 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:10.295 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.919 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:10.284 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.925 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:10.299 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.960 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:10.338 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.962 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:10.317 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:57.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.969+0000 7f15327ab640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:57.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.969+0000 7f15327ab640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:57.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.969+0000 7f15327ab640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:07:57.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:57.969+0000 7f15327ab640 1 -- 172.21.15.37:0/2636531935 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f152c0627b0 con 0x7f152c14be90 2024-03-25T18:07:58.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.013+0000 7f1528ff9640 1 -- 172.21.15.37:0/2636531935 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f15240510c0 con 0x7f152c14be90 2024-03-25T18:07:58.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.013+0000 7f1528ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:07:58.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.013+0000 7f1528ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:07:58.014 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:07:58.014 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:07:58.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.014+0000 7f15327ab640 10 librados: watch_flush enter 2024-03-25T18:07:58.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 10 librados: watch_flush exit 2024-03-25T18:07:58.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 20 client.4204.objecter shutdown clearing up homeless session... 2024-03-25T18:07:58.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 10 client.4204.objecter successfully canceled tick 2024-03-25T18:07:58.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 1 -- 172.21.15.37:0/2636531935 >> v1:172.21.15.117:6800/3703993921 conn(0x7f151806e160 legacy=0x7f1518070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:58.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 10 monclient: shutdown 2024-03-25T18:07:58.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:07:58.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 1 -- 172.21.15.37:0/2636531935 >> v1:172.21.15.117:6789/0 conn(0x7f152c14be90 legacy=0x7f152c142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:07:58.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 1 -- 172.21.15.37:0/2636531935 shutdown_connections 2024-03-25T18:07:58.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 1 -- 172.21.15.37:0/2636531935 >> 172.21.15.37:0/2636531935 conn(0x7f152c0617a0 msgr2=0x7f152c13f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:07:58.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 1 -- 172.21.15.37:0/2636531935 shutdown_connections 2024-03-25T18:07:58.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 1 -- 172.21.15.37:0/2636531935 wait complete. 2024-03-25T18:07:58.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:07:58.015+0000 7f15327ab640 1 librados: shutdown 2024-03-25T18:07:58.017 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:10.390 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.025 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:07:58.029 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:10.392 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.190 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:10.555 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.194 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:10.568 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.216 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:10.571 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.227 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:10.606 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.270 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:10.625 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.293 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:10.656 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.386 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:10.766 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.440 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:10.820 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.468 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:10.833 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.472 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:10.846 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.511 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:10.890 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.538 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:10.893 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.592 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:10.948 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.620 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:10.995 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.675 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:11.049 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.743 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:11.108 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.774 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:11.137 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.777 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:11.155 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.830 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:11.193 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:58.958 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:11.332 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:59.008 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:11.373 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:59.013 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:11.388 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:59.255 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:11.633 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:59.310 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:11.689 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:59.547 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:11.913 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:07:59.603 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:11.969 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:03.688 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:16.043 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:03.743 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:16.117 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:03.970 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:16.325 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:04.197 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:16.577 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:04.690 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:17.053 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:04.815 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:17.189 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:04.832 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:04.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.930+0000 7f281311c640 10 monclient: get_monmap_and_config 2024-03-25T18:08:04.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.930+0000 7f281311c640 10 monclient: build_initial_monmap 2024-03-25T18:08:04.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.930+0000 7f281311c640 10 monclient: monmap: 2024-03-25T18:08:04.931 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:04.931 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:04.931 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:08:04.931746+0000 2024-03-25T18:08:04.932 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:08:04.931746+0000 2024-03-25T18:08:04.932 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:04.932 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:04.932 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:04.932 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:08:04.932 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:08:04.932 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:04.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.931+0000 7f281311c640 1 Processor -- start 2024-03-25T18:08:04.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.931+0000 7f281311c640 1 -- start start 2024-03-25T18:08:04.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.931+0000 7f281311c640 10 monclient: init 2024-03-25T18:08:04.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.932+0000 7f281311c640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:04.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.932+0000 7f281311c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:04.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.932+0000 7f281311c640 10 monclient(hunting): picked mon.noname-a con 0x7f280c06c380 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:04.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.932+0000 7f281311c640 10 monclient(hunting): picked mon.noname-b con 0x7f280c0b12d0 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:04.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.932+0000 7f281311c640 10 monclient(hunting): picked mon.noname-c con 0x7f280c0aedf0 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:04.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.932+0000 7f281311c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280c0b16f0 con 0x7f280c06c380 2024-03-25T18:08:04.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.932+0000 7f281311c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280c0adfe0 con 0x7f280c0aedf0 2024-03-25T18:08:04.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.932+0000 7f281311c640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280c0add00 con 0x7f280c0b12d0 2024-03-25T18:08:04.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.932+0000 7f281311c640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:04.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.932+0000 7f281311c640 10 monclient(hunting): authenticate will time out at 6840.185012s 2024-03-25T18:08:04.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.933+0000 7f2810eb8640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f280c06c380 0x7f280c0b0bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48902/0 (socket says 172.21.15.37:48902) 2024-03-25T18:08:04.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.933+0000 7f2810eb8640 1 -- 172.21.15.37:0/2396057095 learned_addr learned my addr 172.21.15.37:0/2396057095 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:04.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3974317169 0 0) 0x7f280c0add00 con 0x7f280c0b12d0 2024-03-25T18:08:04.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:04.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:04.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:04.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:08:04.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f4003520 con 0x7f280c0b12d0 2024-03-25T18:08:04.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 349825827 0 0) 0x7f280c0b16f0 con 0x7f280c06c380 2024-03-25T18:08:04.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:04.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:04.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:04.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:08:04.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.936+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f4003840 con 0x7f280c06c380 2024-03-25T18:08:04.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.937+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2400677398 0 0) 0x7f280c0adfe0 con 0x7f280c0aedf0 2024-03-25T18:08:04.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.937+0000 7f280b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:08:04.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.937+0000 7f280b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:04.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.937+0000 7f280b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:04.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.937+0000 7f280b7fe640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:08:04.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.937+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f280c0b16f0 con 0x7f280c0aedf0 2024-03-25T18:08:04.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1571460554 0 0) 0x7f27f4003520 con 0x7f280c0b12d0 2024-03-25T18:08:04.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 >> v1:172.21.15.117:6789/0 conn(0x7f280c0aedf0 legacy=0x7f280c14e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:04.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 >> v1:172.21.15.37:6789/0 conn(0x7f280c06c380 legacy=0x7f280c0b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:04.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:08:04.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:08:04.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:04.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280c0ae2c0 con 0x7f280c0b12d0 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f2800001670 con 0x7f280c0b12d0 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f280b7fe640 10 monclient: dump: 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:04.943 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:04.944 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:04.944 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:04.944 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:04.944 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:04.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f281311c640 5 monclient: authenticate success, global_id 4208 2024-03-25T18:08:04.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.940+0000 7f281311c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:04.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.942+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f28000017d0 con 0x7f280c0b12d0 2024-03-25T18:08:04.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.942+0000 7f280b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:04.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.942+0000 7f281311c640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:04.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.942+0000 7f281311c640 4 set_mon_vals no callback set 2024-03-25T18:08:04.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.942+0000 7f280b7fe640 1 -- 172.21.15.37:0/2396057095 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f2800001b60 con 0x7f280c0b12d0 2024-03-25T18:08:04.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.942+0000 7f280b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:04.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.942+0000 7f280b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.942+0000 7f280b7fe640 10 monclient: dump: 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.943+0000 7f281311c640 10 monclient: shutdown 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.943+0000 7f281311c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:04.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.943+0000 7f281311c640 1 -- 172.21.15.37:0/2396057095 >> v1:172.21.15.37:6790/0 conn(0x7f280c0b12d0 legacy=0x7f280c0ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.943+0000 7f281311c640 1 -- 172.21.15.37:0/2396057095 shutdown_connections 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.943+0000 7f281311c640 1 -- 172.21.15.37:0/2396057095 >> 172.21.15.37:0/2396057095 conn(0x7f280c0b5d10 msgr2=0x7f280c0b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.943+0000 7f281311c640 1 -- 172.21.15.37:0/2396057095 shutdown_connections 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.943+0000 7f281311c640 1 -- 172.21.15.37:0/2396057095 wait complete. 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.944+0000 7f281311c640 10 monclient: build_initial_monmap 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.944+0000 7f281311c640 10 monclient: monmap: 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:08:04.948 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:04.949 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:08:04.949 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:08:04.949 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:04.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.945+0000 7f281311c640 1 librados: starting msgr at 2024-03-25T18:08:04.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.945+0000 7f281311c640 1 librados: starting objecter 2024-03-25T18:08:04.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.945+0000 7f281311c640 1 Processor -- start 2024-03-25T18:08:04.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.945+0000 7f281311c640 1 -- start start 2024-03-25T18:08:04.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.945+0000 7f281311c640 1 librados: setting wanted keys 2024-03-25T18:08:04.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.945+0000 7f281311c640 1 librados: calling monclient init 2024-03-25T18:08:04.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.945+0000 7f281311c640 10 monclient: init 2024-03-25T18:08:04.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.946+0000 7f281311c640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.946+0000 7f281311c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.946+0000 7f281311c640 10 monclient(hunting): picked mon.noname-b con 0x7f280c06c380 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.946+0000 7f281311c640 10 monclient(hunting): picked mon.noname-c con 0x7f280c0aedf0 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.946+0000 7f281311c640 10 monclient(hunting): picked mon.noname-a con 0x7f280c0b12d0 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.946+0000 7f281311c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280c06c8e0 con 0x7f280c0b12d0 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.946+0000 7f281311c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280c1a3f40 con 0x7f280c06c380 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.946+0000 7f281311c640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f280c1a43d0 con 0x7f280c0aedf0 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.946+0000 7f281311c640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.946+0000 7f281311c640 10 monclient(hunting): authenticate will time out at 6840.199170s 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.947+0000 7f280bfff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f280c0aedf0 0x7f280c1d75f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:51960/0 (socket says 172.21.15.37:51960) 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.947+0000 7f280bfff640 1 -- 172.21.15.37:0/3766599775 learned_addr learned my addr 172.21.15.37:0/3766599775 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:04.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.947+0000 7f28097fa640 10 client.?.objecter ms_handle_connect 0x7f280c0aedf0 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.947+0000 7f28097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.947+0000 7f28097fa640 10 client.?.objecter ms_handle_connect 0x7f280c06c380 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.947+0000 7f28097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.948+0000 7f28097fa640 10 client.?.objecter ms_handle_connect 0x7f280c0b12d0 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.948+0000 7f28097fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.950+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 521099337 0 0) 0x7f280c1a3f40 con 0x7f280c06c380 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.950+0000 7f28097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.950+0000 7f28097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.950+0000 7f28097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.950+0000 7f28097fa640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.950+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27f8004eb0 con 0x7f280c06c380 2024-03-25T18:08:04.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95725515 0 0) 0x7f280c1a43d0 con 0x7f280c0aedf0 2024-03-25T18:08:04.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:04.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:04.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:04.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 10 monclient(hunting): my global_id is 4211 2024-03-25T18:08:04.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f280c1a3f40 con 0x7f280c0aedf0 2024-03-25T18:08:04.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3022382668 0 0) 0x7f280c06c8e0 con 0x7f280c0b12d0 2024-03-25T18:08:04.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 10 monclient(hunting): _negotiate 2024-03-25T18:08:04.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:04.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:04.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:08:04.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.951+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f280c1a43d0 con 0x7f280c0b12d0 2024-03-25T18:08:04.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.956+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2159914329 0 0) 0x7f27f8004eb0 con 0x7f280c06c380 2024-03-25T18:08:04.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.956+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f280c06c8e0 con 0x7f280c06c380 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.960+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2370251368 0 0) 0x7f280c1a3f40 con 0x7f280c0aedf0 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.960+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27f8004eb0 con 0x7f280c0aedf0 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.960+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f2800001b60 con 0x7f280c0aedf0 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.960+0000 7f28097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.960+0000 7f28097fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.960+0000 7f28097fa640 10 monclient(hunting): dump: 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:04.962 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:04.963 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:04.963 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:04.963 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:04.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.960+0000 7f28097fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:04.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.961+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2399693805 0 0) 0x7f280c1a43d0 con 0x7f280c0b12d0 2024-03-25T18:08:04.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.961+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27f8005eb0 con 0x7f280c0b12d0 2024-03-25T18:08:04.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f27fc001d90 con 0x7f280c06c380 2024-03-25T18:08:04.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:04.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:04.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 10 monclient(hunting): dump: 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1319990136 0 0) 0x7f280c06c8e0 con 0x7f280c06c380 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 >> v1:172.21.15.37:6790/0 conn(0x7f280c0aedf0 legacy=0x7f280c1d75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:04.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 >> v1:172.21.15.37:6789/0 conn(0x7f280c0b12d0 legacy=0x7f280c1c4070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 10 monclient: _finish_hunting 0 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 1 monclient: found mon.b 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f280c1a4f60 con 0x7f280c06c380 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 10 monclient: _finish_auth 0 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 10 monclient: _check_auth_tickets 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f28097fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f281311c640 5 monclient: authenticate success, global_id 4210 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f281311c640 10 monclient: _renew_subs 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f281311c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f281311c640 1 -- 172.21.15.37:0/3766599775 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f280c1a5240 con 0x7f280c06c380 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f281311c640 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:04.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f281311c640 10 monclient: _renew_subs 2024-03-25T18:08:04.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f281311c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:04.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f281311c640 1 -- 172.21.15.37:0/3766599775 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f280c1c4960 con 0x7f280c06c380 2024-03-25T18:08:04.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.962+0000 7f281311c640 1 librados: init done 2024-03-25T18:08:04.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.963+0000 7f281311c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:04.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.963+0000 7f281311c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:04.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.963+0000 7f281311c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:04.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:04.964+0000 7f281311c640 1 -- 172.21.15.37:0/3766599775 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f280c0627b0 con 0x7f280c06c380 2024-03-25T18:08:05.174 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:17.552 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:05.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.233+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27fc001a10 con 0x7f280c06c380 2024-03-25T18:08:05.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.233+0000 7f28097fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:05.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.233+0000 7f280affd640 4 set_mon_vals no callback set 2024-03-25T18:08:05.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.274+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f27fc004d80 con 0x7f280c06c380 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.274+0000 7f28097fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.274+0000 7f28097fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.274+0000 7f28097fa640 10 monclient: dump: 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:05.275 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:05.276 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:05.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.276+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f27fc01ae10 con 0x7f280c06c380 2024-03-25T18:08:05.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.277+0000 7f280bfff640 10 monclient: get_auth_request con 0x7f27f806e5e0 auth_method 0 2024-03-25T18:08:05.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.277+0000 7f28097fa640 10 client.4210.objecter ms_handle_connect 0x7f27f806e5e0 2024-03-25T18:08:05.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.282+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f27fc054c20 con 0x7f280c06c380 2024-03-25T18:08:05.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.282+0000 7f28097fa640 10 client.4210.objecter ms_dispatch 0x7f280c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:05.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.282+0000 7f28097fa640 3 client.4210.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:05.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.282+0000 7f28097fa640 3 client.4210.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:05.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.282+0000 7f28097fa640 20 client.4210.objecter dump_active .. 0 homeless 2024-03-25T18:08:05.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.286+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27fc050560 con 0x7f280c06c380 2024-03-25T18:08:05.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.286+0000 7f28097fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:05.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.286+0000 7f28097fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:05.419 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:17.784 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:05.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.656+0000 7f281311c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:05.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.656+0000 7f281311c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:05.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.656+0000 7f281311c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:05.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.656+0000 7f281311c640 1 -- 172.21.15.37:0/3766599775 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f280c072ac0 con 0x7f280c06c380 2024-03-25T18:08:05.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.683+0000 7f28097fa640 1 -- 172.21.15.37:0/3766599775 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f27fc052660 con 0x7f280c06c380 2024-03-25T18:08:05.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.683+0000 7f28097fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:05.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.683+0000 7f28097fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:05.684 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:08:05.684 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:05.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.685+0000 7f281311c640 10 librados: watch_flush enter 2024-03-25T18:08:05.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.685+0000 7f281311c640 10 librados: watch_flush exit 2024-03-25T18:08:05.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.685+0000 7f281311c640 20 client.4210.objecter shutdown clearing up homeless session... 2024-03-25T18:08:05.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.685+0000 7f281311c640 10 client.4210.objecter successfully canceled tick 2024-03-25T18:08:05.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.686+0000 7f281311c640 1 -- 172.21.15.37:0/3766599775 >> v1:172.21.15.117:6800/3703993921 conn(0x7f27f806e5e0 legacy=0x7f27f8070aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:05.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.686+0000 7f281311c640 10 monclient: shutdown 2024-03-25T18:08:05.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.686+0000 7f281311c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:05.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.686+0000 7f281311c640 1 -- 172.21.15.37:0/3766599775 >> v1:172.21.15.117:6789/0 conn(0x7f280c06c380 legacy=0x7f280c1a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:05.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.686+0000 7f281311c640 1 -- 172.21.15.37:0/3766599775 shutdown_connections 2024-03-25T18:08:05.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.686+0000 7f281311c640 1 -- 172.21.15.37:0/3766599775 >> 172.21.15.37:0/3766599775 conn(0x7f280c0b5d10 msgr2=0x7f280c1cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:05.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.686+0000 7f281311c640 1 -- 172.21.15.37:0/3766599775 shutdown_connections 2024-03-25T18:08:05.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.686+0000 7f281311c640 1 -- 172.21.15.37:0/3766599775 wait complete. 2024-03-25T18:08:05.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:05.686+0000 7f281311c640 1 librados: shutdown 2024-03-25T18:08:05.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:08.989 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:21.364 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:08.996 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:21.370 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:09.222 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:21.577 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:09.225 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:21.581 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:09.291 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:21.646 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:09.295 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:21.650 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:09.330 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:21.710 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:09.336 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:21.716 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:10.314 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:22.688 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:10.364 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:22.692 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:10.541 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:22.904 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:10.547 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:22.910 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:10.955 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:23.333 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:10.959 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:23.338 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:11.559 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:23.924 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:11.562 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:23.928 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:12.505 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:12.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.611+0000 7f8550e55640 10 monclient: get_monmap_and_config 2024-03-25T18:08:12.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.611+0000 7f8550e55640 10 monclient: build_initial_monmap 2024-03-25T18:08:12.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.611+0000 7f8550e55640 10 monclient: monmap: 2024-03-25T18:08:12.612 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:12.612 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:12.612 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:08:12.612167+0000 2024-03-25T18:08:12.612 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:08:12.612167+0000 2024-03-25T18:08:12.612 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:12.612 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:12.612 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:12.613 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:08:12.613 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:08:12.613 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:12.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.611+0000 7f8550e55640 1 Processor -- start 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.611+0000 7f8550e55640 1 -- start start 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.611+0000 7f8550e55640 10 monclient: init 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f8550e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f8550e55640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f8550e55640 10 monclient(hunting): picked mon.noname-c con 0x7f854c0b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f8550e55640 10 monclient(hunting): picked mon.noname-a con 0x7f854c0b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f8550e55640 10 monclient(hunting): picked mon.noname-b con 0x7f854c14be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f8550e55640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c0af940 con 0x7f854c0b1730 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f8550e55640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c0b0a70 con 0x7f854c0b4d60 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f8550e55640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c0b0d50 con 0x7f854c14be90 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f8550e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f8550e55640 10 monclient(hunting): authenticate will time out at 6847.865547s 2024-03-25T18:08:12.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f854ad9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f854c14be90 0x7f854c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:32768/0 (socket says 172.21.15.37:32768) 2024-03-25T18:08:12.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.613+0000 7f854ad9d640 1 -- 172.21.15.37:0/1643774620 learned_addr learned my addr 172.21.15.37:0/1643774620 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:12.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.616+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787341525 0 0) 0x7f854c0af940 con 0x7f854c0b1730 2024-03-25T18:08:12.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.616+0000 7f854959a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:12.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.616+0000 7f854959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:12.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.616+0000 7f854959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:12.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.616+0000 7f854959a640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:08:12.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.616+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8534003520 con 0x7f854c0b1730 2024-03-25T18:08:12.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453800545 0 0) 0x7f854c0b0a70 con 0x7f854c0b4d60 2024-03-25T18:08:12.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:12.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:12.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:12.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:08:12.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8534003840 con 0x7f854c0b4d60 2024-03-25T18:08:12.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 810492842 0 0) 0x7f854c0b0d50 con 0x7f854c14be90 2024-03-25T18:08:12.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:12.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:12.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:12.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:08:12.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.617+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8534003b80 con 0x7f854c14be90 2024-03-25T18:08:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1959041066 0 0) 0x7f8534003520 con 0x7f854c0b1730 2024-03-25T18:08:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 >> v1:172.21.15.37:6790/0 conn(0x7f854c14be90 legacy=0x7f854c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 >> v1:172.21.15.117:6789/0 conn(0x7f854c0b4d60 legacy=0x7f854c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 10 monclient: _finish_hunting 0 2024-03-25T18:08:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 1 monclient: found mon.noname-a 2024-03-25T18:08:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:08:12.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c0b1030 con 0x7f854c0b1730 2024-03-25T18:08:12.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 10 monclient: _finish_auth 0 2024-03-25T18:08:12.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 10 monclient: _check_auth_tickets 2024-03-25T18:08:12.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:12.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8540001670 con 0x7f854c0b1730 2024-03-25T18:08:12.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:12.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:12.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f854959a640 10 monclient: dump: 2024-03-25T18:08:12.622 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:12.622 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:12.622 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:12.623 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:12.623 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:12.623 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:12.623 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:12.623 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:12.623 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:12.623 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:12.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f8550e55640 5 monclient: authenticate success, global_id 4212 2024-03-25T18:08:12.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.620+0000 7f8550e55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.623+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85400017d0 con 0x7f854c0b1730 2024-03-25T18:08:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.623+0000 7f854959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.623+0000 7f854959a640 1 -- 172.21.15.37:0/1643774620 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8540001be0 con 0x7f854c0b1730 2024-03-25T18:08:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.623+0000 7f8550e55640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.623+0000 7f854959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.623+0000 7f8550e55640 4 set_mon_vals no callback set 2024-03-25T18:08:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.623+0000 7f854959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:12.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.623+0000 7f854959a640 10 monclient: dump: 2024-03-25T18:08:12.624 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:12.625 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:12.625 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:12.625 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:12.625 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:12.625 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:12.625 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:12.625 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:12.625 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:12.625 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.624+0000 7f8550e55640 10 monclient: shutdown 2024-03-25T18:08:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.624+0000 7f8550e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.624+0000 7f8550e55640 1 -- 172.21.15.37:0/1643774620 >> v1:172.21.15.37:6789/0 conn(0x7f854c0b1730 legacy=0x7f854c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.625+0000 7f8550e55640 1 -- 172.21.15.37:0/1643774620 shutdown_connections 2024-03-25T18:08:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.625+0000 7f8550e55640 1 -- 172.21.15.37:0/1643774620 >> 172.21.15.37:0/1643774620 conn(0x7f854c0617a0 msgr2=0x7f854c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.625+0000 7f8550e55640 1 -- 172.21.15.37:0/1643774620 shutdown_connections 2024-03-25T18:08:12.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.625+0000 7f8550e55640 1 -- 172.21.15.37:0/1643774620 wait complete. 2024-03-25T18:08:12.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.626+0000 7f8550e55640 10 monclient: build_initial_monmap 2024-03-25T18:08:12.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.626+0000 7f8550e55640 10 monclient: monmap: 2024-03-25T18:08:12.627 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:12.627 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:12.627 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:08:12.627 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.626+0000 7f8550e55640 1 librados: starting msgr at 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.626+0000 7f8550e55640 1 librados: starting objecter 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.627+0000 7f8550e55640 1 Processor -- start 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.627+0000 7f8550e55640 1 -- start start 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.627+0000 7f8550e55640 1 librados: setting wanted keys 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.627+0000 7f8550e55640 1 librados: calling monclient init 2024-03-25T18:08:12.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.627+0000 7f8550e55640 10 monclient: init 2024-03-25T18:08:12.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.627+0000 7f8550e55640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:12.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f8550e55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f8550e55640 10 monclient(hunting): picked mon.noname-b con 0x7f854c0b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f8550e55640 10 monclient(hunting): picked mon.noname-c con 0x7f854c0b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f8550e55640 10 monclient(hunting): picked mon.noname-a con 0x7f854c14be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f8550e55640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c144330 con 0x7f854c14be90 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f8550e55640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c1453c0 con 0x7f854c0b1730 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f8550e55640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c13ee30 con 0x7f854c0b4d60 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f8550e55640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f8550e55640 10 monclient(hunting): authenticate will time out at 6847.881091s 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f8549d9b640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f854c0b4d60 0x7f854c1dada0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:32776/0 (socket says 172.21.15.37:32776) 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f854a59c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f854c0b1730 0x7f854c1427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:50906/0 (socket says 172.21.15.37:50906) 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.628+0000 7f854a59c640 1 -- 172.21.15.37:0/2253697693 learned_addr learned my addr 172.21.15.37:0/2253697693 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.629+0000 7f853affd640 10 client.?.objecter ms_handle_connect 0x7f854c0b1730 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.629+0000 7f853affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.629+0000 7f853affd640 10 client.?.objecter ms_handle_connect 0x7f854c0b4d60 2024-03-25T18:08:12.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.629+0000 7f853affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:12.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.629+0000 7f853affd640 10 client.?.objecter ms_handle_connect 0x7f854c14be90 2024-03-25T18:08:12.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.629+0000 7f853affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:12.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156915063 0 0) 0x7f854c13ee30 con 0x7f854c0b4d60 2024-03-25T18:08:12.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:12.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:12.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:12.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:08:12.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f852c004eb0 con 0x7f854c0b4d60 2024-03-25T18:08:12.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1572209858 0 0) 0x7f854c1453c0 con 0x7f854c0b1730 2024-03-25T18:08:12.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:12.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:12.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:12.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:08:12.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.631+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f852c006230 con 0x7f854c0b1730 2024-03-25T18:08:12.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.632+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580398513 0 0) 0x7f854c144330 con 0x7f854c14be90 2024-03-25T18:08:12.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.632+0000 7f853affd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:12.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.632+0000 7f853affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:12.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.632+0000 7f853affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:12.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.632+0000 7f853affd640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:08:12.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.632+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f854c1453c0 con 0x7f854c14be90 2024-03-25T18:08:12.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.636+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2663198046 0 0) 0x7f852c004eb0 con 0x7f854c0b4d60 2024-03-25T18:08:12.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.636+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f854c144330 con 0x7f854c0b4d60 2024-03-25T18:08:12.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.636+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8540001ae0 con 0x7f854c0b4d60 2024-03-25T18:08:12.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.636+0000 7f853affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:12.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.636+0000 7f853affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:12.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.636+0000 7f853affd640 10 monclient(hunting): dump: 2024-03-25T18:08:12.637 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:12.637 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:12.637 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:12.638 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:12.638 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:12.638 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:12.638 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:12.638 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:12.638 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:12.638 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:12.638 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.636+0000 7f853affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2615362967 0 0) 0x7f852c006230 con 0x7f854c0b1730 2024-03-25T18:08:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f852c006d30 con 0x7f854c0b1730 2024-03-25T18:08:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f853c0019f0 con 0x7f854c0b1730 2024-03-25T18:08:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:12.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 10 monclient(hunting): dump: 2024-03-25T18:08:12.642 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:12.642 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:12.642 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:12.642 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1702731261 0 0) 0x7f854c144330 con 0x7f854c0b4d60 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 >> v1:172.21.15.117:6789/0 conn(0x7f854c0b1730 legacy=0x7f854c1427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 >> v1:172.21.15.37:6789/0 conn(0x7f854c14be90 legacy=0x7f854c1db4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 1 monclient: found mon.c 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:08:12.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c143900 con 0x7f854c0b4d60 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 10 monclient: _finish_auth 0 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f853affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f8550e55640 5 monclient: authenticate success, global_id 4217 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f8550e55640 10 monclient: _renew_subs 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f8550e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f8550e55640 1 -- 172.21.15.37:0/2253697693 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c13ea50 con 0x7f854c0b4d60 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f8550e55640 10 client.4217.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f8550e55640 10 monclient: _renew_subs 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f8550e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f8550e55640 1 -- 172.21.15.37:0/2253697693 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c131eb0 con 0x7f854c0b4d60 2024-03-25T18:08:12.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.639+0000 7f8550e55640 1 librados: init done 2024-03-25T18:08:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.641+0000 7f8550e55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.641+0000 7f8550e55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.641+0000 7f8550e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:08:12.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.641+0000 7f8550e55640 1 -- 172.21.15.37:0/2253697693 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c131eb0 con 0x7f854c0b4d60 2024-03-25T18:08:12.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.771+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85400011f0 con 0x7f854c0b4d60 2024-03-25T18:08:12.772 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.771+0000 7f853affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.771+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8540001ae0 con 0x7f854c0b4d60 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.771+0000 7f853affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.771+0000 7f853affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.771+0000 7f853affd640 10 monclient: dump: 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:12.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.771+0000 7f8548d99640 4 set_mon_vals no callback set 2024-03-25T18:08:12.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.773+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f8540003920 con 0x7f854c0b4d60 2024-03-25T18:08:12.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.773+0000 7f854a59c640 10 monclient: get_auth_request con 0x7f852c06e600 auth_method 0 2024-03-25T18:08:12.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.773+0000 7f853affd640 10 client.4217.objecter ms_handle_connect 0x7f852c06e600 2024-03-25T18:08:12.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.829+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f85400544a0 con 0x7f854c0b4d60 2024-03-25T18:08:12.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.829+0000 7f853affd640 10 client.4217.objecter ms_dispatch 0x7f854c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:12.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.829+0000 7f853affd640 3 client.4217.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:12.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.829+0000 7f853affd640 3 client.4217.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:12.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.829+0000 7f853affd640 20 client.4217.objecter dump_active .. 0 homeless 2024-03-25T18:08:12.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.929+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f854004fde0 con 0x7f854c0b4d60 2024-03-25T18:08:12.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.929+0000 7f853affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:12.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:12.929+0000 7f853affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:13.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.214+0000 7f8550e55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:13.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.214+0000 7f8550e55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:13.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.214+0000 7f8550e55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:08:13.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.214+0000 7f8550e55640 1 -- 172.21.15.37:0/2253697693 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f854c131eb0 con 0x7f854c0b4d60 2024-03-25T18:08:13.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.284+0000 7f853affd640 1 -- 172.21.15.37:0/2253697693 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f8540051ee0 con 0x7f854c0b4d60 2024-03-25T18:08:13.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.284+0000 7f853affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:13.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.284+0000 7f853affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:13.285 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:08:13.286 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:13.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.287+0000 7f8550e55640 10 librados: watch_flush enter 2024-03-25T18:08:13.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.287+0000 7f8550e55640 10 librados: watch_flush exit 2024-03-25T18:08:13.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.287+0000 7f8550e55640 20 client.4217.objecter shutdown clearing up homeless session... 2024-03-25T18:08:13.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.287+0000 7f8550e55640 10 client.4217.objecter successfully canceled tick 2024-03-25T18:08:13.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.287+0000 7f8550e55640 1 -- 172.21.15.37:0/2253697693 >> v1:172.21.15.117:6800/3703993921 conn(0x7f852c06e600 legacy=0x7f852c070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:13.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.287+0000 7f8550e55640 10 monclient: shutdown 2024-03-25T18:08:13.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.287+0000 7f8550e55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:13.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.287+0000 7f8550e55640 1 -- 172.21.15.37:0/2253697693 >> v1:172.21.15.37:6790/0 conn(0x7f854c0b4d60 legacy=0x7f854c1dada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:13.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.287+0000 7f8550e55640 1 -- 172.21.15.37:0/2253697693 shutdown_connections 2024-03-25T18:08:13.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.287+0000 7f8550e55640 1 -- 172.21.15.37:0/2253697693 >> 172.21.15.37:0/2253697693 conn(0x7f854c0617a0 msgr2=0x7f854c0b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:13.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.288+0000 7f8550e55640 1 -- 172.21.15.37:0/2253697693 shutdown_connections 2024-03-25T18:08:13.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.288+0000 7f8550e55640 1 -- 172.21.15.37:0/2253697693 wait complete. 2024-03-25T18:08:13.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:13.288+0000 7f8550e55640 1 librados: shutdown 2024-03-25T18:08:13.300 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:13.396 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:00:25.770 35534== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:13.615 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:00:25.970 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:13.674 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:00:26.054 35533== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:13.896 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:00:26.251 35535== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:15.274 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:00:27.637 35479== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:15.295 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:00:27.669 35477== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:15.458 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:00:27.837 35476== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:16.008 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:00:28.373 35478== Warning: unimplemented fcntl command: 1038 2024-03-25T18:08:20.107 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:20.204 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.203+0000 7f7cc482e640 10 monclient: get_monmap_and_config 2024-03-25T18:08:20.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.203+0000 7f7cc482e640 10 monclient: build_initial_monmap 2024-03-25T18:08:20.205 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.203+0000 7f7cc482e640 10 monclient: monmap: 2024-03-25T18:08:20.205 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:20.205 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:20.205 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:08:20.204774+0000 2024-03-25T18:08:20.205 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:08:20.204774+0000 2024-03-25T18:08:20.205 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:20.206 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:20.206 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:20.206 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:08:20.206 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:08:20.206 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:20.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.204+0000 7f7cc482e640 1 Processor -- start 2024-03-25T18:08:20.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.204+0000 7f7cc482e640 1 -- start start 2024-03-25T18:08:20.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.204+0000 7f7cc482e640 10 monclient: init 2024-03-25T18:08:20.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.205+0000 7f7cc482e640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:20.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.205+0000 7f7cc482e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:20.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.205+0000 7f7cc482e640 10 monclient(hunting): picked mon.noname-b con 0x7f7cc00b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:20.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.205+0000 7f7cc482e640 10 monclient(hunting): picked mon.noname-a con 0x7f7cc00b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:20.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.205+0000 7f7cc482e640 10 monclient(hunting): picked mon.noname-c con 0x7f7cc014be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:20.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.205+0000 7f7cc482e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc00af940 con 0x7f7cc00b1730 2024-03-25T18:08:20.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.205+0000 7f7cc482e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc00b0a70 con 0x7f7cc014be90 2024-03-25T18:08:20.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.205+0000 7f7cc482e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc00b0d50 con 0x7f7cc00b4d60 2024-03-25T18:08:20.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.205+0000 7f7cc482e640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:20.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.205+0000 7f7cc482e640 10 monclient(hunting): authenticate will time out at 6855.458137s 2024-03-25T18:08:20.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.206+0000 7f7cbe59c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f7cc00b4d60 0x7f7cc013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:46666/0 (socket says 172.21.15.37:46666) 2024-03-25T18:08:20.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.206+0000 7f7cbe59c640 1 -- 172.21.15.37:0/3039435713 learned_addr learned my addr 172.21.15.37:0/3039435713 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:20.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.209+0000 7f7cbd59a640 1 -- 172.21.15.37:0/3039435713 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4048659719 0 0) 0x7f7cc00b0a70 con 0x7f7cc014be90 2024-03-25T18:08:20.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.209+0000 7f7cbd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.209+0000 7f7cbd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.210+0000 7f7cbd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.210+0000 7f7cbd59a640 10 monclient(hunting): my global_id is 4219 2024-03-25T18:08:20.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.210+0000 7f7cbd59a640 1 -- 172.21.15.37:0/3039435713 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ca8003520 con 0x7f7cc014be90 2024-03-25T18:08:20.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.210+0000 7f7cbd59a640 1 -- 172.21.15.37:0/3039435713 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149726581 0 0) 0x7f7cc00b0d50 con 0x7f7cc00b4d60 2024-03-25T18:08:20.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.210+0000 7f7cbd59a640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.210+0000 7f7cbd59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.210+0000 7f7cbd59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.210+0000 7f7cbd59a640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:08:20.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.210+0000 7f7cbd59a640 1 -- 172.21.15.37:0/3039435713 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc00b0a70 con 0x7f7cc00b4d60 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 1 -- 172.21.15.37:0/3039435713 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1743292398 0 0) 0x7f7ca8003520 con 0x7f7cc014be90 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 1 -- 172.21.15.37:0/3039435713 >> v1:172.21.15.37:6790/0 conn(0x7f7cc00b4d60 legacy=0x7f7cc013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 1 -- 172.21.15.37:0/3039435713 >> v1:172.21.15.37:6789/0 conn(0x7f7cc00b1730 legacy=0x7f7cc00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 10 monclient: _finish_hunting 0 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 1 monclient: found mon.noname-c 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 1 -- 172.21.15.37:0/3039435713 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc00b1030 con 0x7f7cc014be90 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 10 monclient: _finish_auth 0 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 10 monclient: _check_auth_tickets 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 1 -- 172.21.15.37:0/3039435713 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7cb8001a10 con 0x7f7cc014be90 2024-03-25T18:08:20.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cbd59a640 10 monclient: dump: 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.214+0000 7f7cc482e640 5 monclient: authenticate success, global_id 4219 2024-03-25T18:08:20.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.215+0000 7f7cc482e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:20.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.216+0000 7f7cbd59a640 1 -- 172.21.15.37:0/3039435713 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cb8001b70 con 0x7f7cc014be90 2024-03-25T18:08:20.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.216+0000 7f7cbd59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:20.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.216+0000 7f7cc482e640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:20.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.216+0000 7f7cc482e640 4 set_mon_vals no callback set 2024-03-25T18:08:20.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.217+0000 7f7cbd59a640 1 -- 172.21.15.37:0/3039435713 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7cb8001f80 con 0x7f7cc014be90 2024-03-25T18:08:20.219 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.217+0000 7f7cbd59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.217+0000 7f7cbd59a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.217+0000 7f7cbd59a640 10 monclient: dump: 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:20.220 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:20.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.217+0000 7f7cc482e640 10 monclient: shutdown 2024-03-25T18:08:20.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.217+0000 7f7cc482e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:20.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.217+0000 7f7cc482e640 1 -- 172.21.15.37:0/3039435713 >> v1:172.21.15.117:6789/0 conn(0x7f7cc014be90 legacy=0x7f7cc014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.218+0000 7f7cc482e640 1 -- 172.21.15.37:0/3039435713 shutdown_connections 2024-03-25T18:08:20.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.218+0000 7f7cc482e640 1 -- 172.21.15.37:0/3039435713 >> 172.21.15.37:0/3039435713 conn(0x7f7cc00617a0 msgr2=0x7f7cc0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:20.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.218+0000 7f7cc482e640 1 -- 172.21.15.37:0/3039435713 shutdown_connections 2024-03-25T18:08:20.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.218+0000 7f7cc482e640 1 -- 172.21.15.37:0/3039435713 wait complete. 2024-03-25T18:08:20.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.219+0000 7f7cc482e640 10 monclient: build_initial_monmap 2024-03-25T18:08:20.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.219+0000 7f7cc482e640 10 monclient: monmap: 2024-03-25T18:08:20.223 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:20.223 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.219+0000 7f7cc482e640 1 librados: starting msgr at 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.219+0000 7f7cc482e640 1 librados: starting objecter 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.220+0000 7f7cc482e640 1 Processor -- start 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.220+0000 7f7cc482e640 1 -- start start 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.220+0000 7f7cc482e640 1 librados: setting wanted keys 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.220+0000 7f7cc482e640 1 librados: calling monclient init 2024-03-25T18:08:20.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.220+0000 7f7cc482e640 10 monclient: init 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.220+0000 7f7cc482e640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.220+0000 7f7cc482e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.220+0000 7f7cc482e640 10 monclient(hunting): picked mon.noname-a con 0x7f7cc00b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.221+0000 7f7cc482e640 10 monclient(hunting): picked mon.noname-b con 0x7f7cc00b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.221+0000 7f7cc482e640 10 monclient(hunting): picked mon.noname-c con 0x7f7cc014be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.221+0000 7f7cc482e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc01408a0 con 0x7f7cc00b1730 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.221+0000 7f7cc482e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc01374e0 con 0x7f7cc00b4d60 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.221+0000 7f7cc482e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cc0140380 con 0x7f7cc014be90 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.221+0000 7f7cc482e640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.221+0000 7f7cc482e640 10 monclient(hunting): authenticate will time out at 6855.473577s 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.221+0000 7f7cbed9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f7cc014be90 0x7f7cc01db1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:46672/0 (socket says 172.21.15.37:46672) 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.221+0000 7f7cbed9d640 1 -- 172.21.15.37:0/4130079835 learned_addr learned my addr 172.21.15.37:0/4130079835 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.221+0000 7f7caeffd640 10 client.?.objecter ms_handle_connect 0x7f7cc014be90 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.221+0000 7f7caeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.222+0000 7f7caeffd640 10 client.?.objecter ms_handle_connect 0x7f7cc00b1730 2024-03-25T18:08:20.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.222+0000 7f7caeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.222+0000 7f7caeffd640 10 client.?.objecter ms_handle_connect 0x7f7cc00b4d60 2024-03-25T18:08:20.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.222+0000 7f7caeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 85366619 0 0) 0x7f7cc01408a0 con 0x7f7cc00b1730 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ca0004eb0 con 0x7f7cc00b1730 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168872644 0 0) 0x7f7cc01374e0 con 0x7f7cc00b4d60 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): my global_id is 4222 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc01408a0 con 0x7f7cc00b4d60 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628944613 0 0) 0x7f7cc0140380 con 0x7f7cc014be90 2024-03-25T18:08:20.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:20.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:20.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:20.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:08:20.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.225+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7cc01374e0 con 0x7f7cc014be90 2024-03-25T18:08:20.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.230+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 420379433 0 0) 0x7f7ca0004eb0 con 0x7f7cc00b1730 2024-03-25T18:08:20.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.230+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cc0140380 con 0x7f7cc00b1730 2024-03-25T18:08:20.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.230+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7cb0001930 con 0x7f7cc00b1730 2024-03-25T18:08:20.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.230+0000 7f7caeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.230+0000 7f7caeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:20.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.230+0000 7f7caeffd640 10 monclient(hunting): dump: 2024-03-25T18:08:20.231 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:20.231 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:20.231 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:20.232 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:20.232 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.232 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:20.232 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:20.232 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:20.232 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:20.232 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:20.232 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.230+0000 7f7caeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:20.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.232+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3847850004 0 0) 0x7f7cc01374e0 con 0x7f7cc014be90 2024-03-25T18:08:20.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.232+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ca0006b80 con 0x7f7cc014be90 2024-03-25T18:08:20.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.232+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 243013981 0 0) 0x7f7cc01408a0 con 0x7f7cc00b4d60 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.232+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7cc01374e0 con 0x7f7cc00b4d60 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.232+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7cb4001790 con 0x7f7cc00b4d60 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.232+0000 7f7caeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.232+0000 7f7caeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.232+0000 7f7caeffd640 10 monclient(hunting): dump: 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:20.234 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:20.235 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:20.235 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:20.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.233+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 193601015 0 0) 0x7f7cc0140380 con 0x7f7cc00b1730 2024-03-25T18:08:20.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.233+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 >> v1:172.21.15.37:6790/0 conn(0x7f7cc014be90 legacy=0x7f7cc01db1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.233+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 >> v1:172.21.15.117:6789/0 conn(0x7f7cc00b4d60 legacy=0x7f7cc01daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.233+0000 7f7caeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:20.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.233+0000 7f7caeffd640 1 monclient: found mon.a 2024-03-25T18:08:20.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.233+0000 7f7caeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:20.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.233+0000 7f7caeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.233+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cc0140510 con 0x7f7cc00b1730 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.233+0000 7f7caeffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.233+0000 7f7caeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.233+0000 7f7caeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.234+0000 7f7cc482e640 5 monclient: authenticate success, global_id 4218 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.234+0000 7f7cc482e640 10 monclient: _renew_subs 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.234+0000 7f7cc482e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.234+0000 7f7cc482e640 1 -- 172.21.15.37:0/4130079835 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cc0132520 con 0x7f7cc00b1730 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.234+0000 7f7cc482e640 10 client.4218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.234+0000 7f7cc482e640 10 monclient: _renew_subs 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.234+0000 7f7cc482e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.234+0000 7f7cc482e640 1 -- 172.21.15.37:0/4130079835 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cc0137e30 con 0x7f7cc00b1730 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.234+0000 7f7cc482e640 1 librados: init done 2024-03-25T18:08:20.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.235+0000 7f7cc482e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.235+0000 7f7cc482e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.235+0000 7f7cc482e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:08:20.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.235+0000 7f7cc482e640 1 -- 172.21.15.37:0/4130079835 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cc0140700 con 0x7f7cc00b1730 2024-03-25T18:08:20.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.252+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7cb00047e0 con 0x7f7cc00b1730 2024-03-25T18:08:20.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.252+0000 7f7caeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:20.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.252+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7cb0001930 con 0x7f7cc00b1730 2024-03-25T18:08:20.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.252+0000 7f7caeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:20.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.252+0000 7f7caeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.252+0000 7f7caeffd640 10 monclient: dump: 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:20.254 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.252+0000 7f7cbcd99640 4 set_mon_vals no callback set 2024-03-25T18:08:20.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.253+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f7cb0004ad0 con 0x7f7cc00b1730 2024-03-25T18:08:20.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.254+0000 7f7cbdd9b640 10 monclient: get_auth_request con 0x7f7ca006e7c0 auth_method 0 2024-03-25T18:08:20.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.254+0000 7f7caeffd640 10 client.4218.objecter ms_handle_connect 0x7f7ca006e7c0 2024-03-25T18:08:20.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.259+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f7cb0001750 con 0x7f7cc00b1730 2024-03-25T18:08:20.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.259+0000 7f7caeffd640 10 client.4218.objecter ms_dispatch 0x7f7cc0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:20.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.259+0000 7f7caeffd640 3 client.4218.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:20.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.259+0000 7f7caeffd640 3 client.4218.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:20.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.259+0000 7f7caeffd640 20 client.4218.objecter dump_active .. 0 homeless 2024-03-25T18:08:20.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.514+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7cb004fd20 con 0x7f7cc00b1730 2024-03-25T18:08:20.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.514+0000 7f7caeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:20.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.514+0000 7f7caeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.773+0000 7f7cc482e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.773+0000 7f7cc482e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.773+0000 7f7cc482e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:08:20.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.773+0000 7f7cc482e640 1 -- 172.21.15.37:0/4130079835 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7cc01406a0 con 0x7f7cc00b1730 2024-03-25T18:08:20.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.788+0000 7f7caeffd640 1 -- 172.21.15.37:0/4130079835 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f7cb0051c50 con 0x7f7cc00b1730 2024-03-25T18:08:20.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.788+0000 7f7caeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:20.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.788+0000 7f7caeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:20.789 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:08:20.789 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:20.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 10 librados: watch_flush enter 2024-03-25T18:08:20.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 10 librados: watch_flush exit 2024-03-25T18:08:20.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 20 client.4218.objecter shutdown clearing up homeless session... 2024-03-25T18:08:20.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 10 client.4218.objecter successfully canceled tick 2024-03-25T18:08:20.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 1 -- 172.21.15.37:0/4130079835 >> v1:172.21.15.117:6800/3703993921 conn(0x7f7ca006e7c0 legacy=0x7f7ca0070c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 10 monclient: shutdown 2024-03-25T18:08:20.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:20.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 1 -- 172.21.15.37:0/4130079835 >> v1:172.21.15.37:6789/0 conn(0x7f7cc00b1730 legacy=0x7f7cc0131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:20.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 1 -- 172.21.15.37:0/4130079835 shutdown_connections 2024-03-25T18:08:20.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 1 -- 172.21.15.37:0/4130079835 >> 172.21.15.37:0/4130079835 conn(0x7f7cc00617a0 msgr2=0x7f7cc013c150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:20.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 1 -- 172.21.15.37:0/4130079835 shutdown_connections 2024-03-25T18:08:20.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 1 -- 172.21.15.37:0/4130079835 wait complete. 2024-03-25T18:08:20.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:20.791+0000 7f7cc482e640 1 librados: shutdown 2024-03-25T18:08:20.803 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:22.017 INFO:tasks.ceph.osd.7.smithi117.stderr:2024-03-25T18:08:22.005+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:08:22.168 INFO:tasks.ceph.osd.6.smithi117.stderr:2024-03-25T18:08:22.164+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:08:22.349 INFO:tasks.ceph.osd.5.smithi117.stderr:2024-03-25T18:08:22.298+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:08:22.472 INFO:tasks.ceph.osd.4.smithi117.stderr:2024-03-25T18:08:22.468+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:08:24.079 INFO:tasks.ceph.osd.3.smithi037.stderr:2024-03-25T18:08:24.015+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:08:24.240 INFO:tasks.ceph.osd.1.smithi037.stderr:2024-03-25T18:08:24.235+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:08:24.829 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-03-25T18:08:24.824+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:08:25.631 INFO:tasks.ceph.osd.2.smithi037.stderr:2024-03-25T18:08:25.626+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:08:27.609 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:27.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.722+0000 7fa2d23ed640 10 monclient: get_monmap_and_config 2024-03-25T18:08:27.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.722+0000 7fa2d23ed640 10 monclient: build_initial_monmap 2024-03-25T18:08:27.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.722+0000 7fa2d23ed640 10 monclient: monmap: 2024-03-25T18:08:27.723 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:27.723 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:27.723 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:08:27.723379+0000 2024-03-25T18:08:27.723 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:08:27.723379+0000 2024-03-25T18:08:27.723 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:27.724 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:27.724 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:27.724 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:08:27.724 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:08:27.724 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:27.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.723+0000 7fa2d23ed640 1 Processor -- start 2024-03-25T18:08:27.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.723+0000 7fa2d23ed640 1 -- start start 2024-03-25T18:08:27.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.723+0000 7fa2d23ed640 10 monclient: init 2024-03-25T18:08:27.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.724+0000 7fa2d23ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:27.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.724+0000 7fa2d23ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:27.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.724+0000 7fa2d23ed640 10 monclient(hunting): picked mon.noname-a con 0x7fa2cc0b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:27.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.724+0000 7fa2d23ed640 10 monclient(hunting): picked mon.noname-b con 0x7fa2cc0b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:27.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.724+0000 7fa2d23ed640 10 monclient(hunting): picked mon.noname-c con 0x7fa2cc14be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:27.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.724+0000 7fa2d23ed640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2cc0af940 con 0x7fa2cc0b4d60 2024-03-25T18:08:27.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.724+0000 7fa2d23ed640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2cc0b0a70 con 0x7fa2cc14be90 2024-03-25T18:08:27.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.724+0000 7fa2d23ed640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2cc0b0d50 con 0x7fa2cc0b1730 2024-03-25T18:08:27.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.724+0000 7fa2d23ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:27.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.724+0000 7fa2d23ed640 10 monclient(hunting): authenticate will time out at 6862.976407s 2024-03-25T18:08:27.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.737+0000 7fa2cbfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa2cc0b4d60 0x7fa2cc13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:54168/0 (socket says 172.21.15.37:54168) 2024-03-25T18:08:27.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.737+0000 7fa2cbfff640 1 -- 172.21.15.37:0/3900089625 learned_addr learned my addr 172.21.15.37:0/3900089625 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:27.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.760+0000 7fa2caffd640 1 -- 172.21.15.37:0/3900089625 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3003829656 0 0) 0x7fa2cc0b0a70 con 0x7fa2cc14be90 2024-03-25T18:08:27.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.760+0000 7fa2caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:27.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.760+0000 7fa2caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:27.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.760+0000 7fa2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:27.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.760+0000 7fa2caffd640 10 monclient(hunting): my global_id is 4249 2024-03-25T18:08:27.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.760+0000 7fa2caffd640 1 -- 172.21.15.37:0/3900089625 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2b4003520 con 0x7fa2cc14be90 2024-03-25T18:08:27.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.763+0000 7fa2caffd640 1 -- 172.21.15.37:0/3900089625 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151476817 0 0) 0x7fa2cc0af940 con 0x7fa2cc0b4d60 2024-03-25T18:08:27.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.763+0000 7fa2caffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:27.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.763+0000 7fa2caffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:27.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.763+0000 7fa2caffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:27.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.763+0000 7fa2caffd640 10 monclient(hunting): my global_id is 4245 2024-03-25T18:08:27.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.763+0000 7fa2caffd640 1 -- 172.21.15.37:0/3900089625 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2cc0b0a70 con 0x7fa2cc0b4d60 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 1 -- 172.21.15.37:0/3900089625 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1647538979 0 0) 0x7fa2cc0b0a70 con 0x7fa2cc0b4d60 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 1 -- 172.21.15.37:0/3900089625 >> v1:172.21.15.37:6790/0 conn(0x7fa2cc0b1730 legacy=0x7fa2cc0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 1 -- 172.21.15.37:0/3900089625 >> v1:172.21.15.117:6789/0 conn(0x7fa2cc14be90 legacy=0x7fa2cc14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 1 monclient: found mon.noname-a 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 1 -- 172.21.15.37:0/3900089625 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc0b1030 con 0x7fa2cc0b4d60 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:27.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 1 -- 172.21.15.37:0/3900089625 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa2bc001670 con 0x7fa2cc0b4d60 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2caffd640 10 monclient: dump: 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2d23ed640 5 monclient: authenticate success, global_id 4245 2024-03-25T18:08:27.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.788+0000 7fa2d23ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:27.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.884+0000 7fa2caffd640 1 -- 172.21.15.37:0/3900089625 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2bc0017d0 con 0x7fa2cc0b4d60 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.884+0000 7fa2caffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.884+0000 7fa2caffd640 1 -- 172.21.15.37:0/3900089625 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa2bc001ba0 con 0x7fa2cc0b4d60 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.884+0000 7fa2d23ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.884+0000 7fa2d23ed640 4 set_mon_vals no callback set 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.884+0000 7fa2caffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.884+0000 7fa2caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.884+0000 7fa2caffd640 10 monclient: dump: 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:27.885 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:27.886 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:27.886 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:27.886 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:27.886 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:27.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.885+0000 7fa2d23ed640 10 monclient: shutdown 2024-03-25T18:08:27.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.885+0000 7fa2d23ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:27.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.885+0000 7fa2d23ed640 1 -- 172.21.15.37:0/3900089625 >> v1:172.21.15.37:6789/0 conn(0x7fa2cc0b4d60 legacy=0x7fa2cc13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:27.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.885+0000 7fa2d23ed640 1 -- 172.21.15.37:0/3900089625 shutdown_connections 2024-03-25T18:08:27.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.885+0000 7fa2d23ed640 1 -- 172.21.15.37:0/3900089625 >> 172.21.15.37:0/3900089625 conn(0x7fa2cc0617a0 msgr2=0x7fa2cc061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:27.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.885+0000 7fa2d23ed640 1 -- 172.21.15.37:0/3900089625 shutdown_connections 2024-03-25T18:08:27.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.885+0000 7fa2d23ed640 1 -- 172.21.15.37:0/3900089625 wait complete. 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.886+0000 7fa2d23ed640 10 monclient: build_initial_monmap 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.886+0000 7fa2d23ed640 10 monclient: monmap: 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.886+0000 7fa2d23ed640 1 librados: starting msgr at 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.886+0000 7fa2d23ed640 1 librados: starting objecter 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.886+0000 7fa2d23ed640 1 Processor -- start 2024-03-25T18:08:27.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.886+0000 7fa2d23ed640 1 -- start start 2024-03-25T18:08:27.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.886+0000 7fa2d23ed640 1 librados: setting wanted keys 2024-03-25T18:08:27.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.886+0000 7fa2d23ed640 1 librados: calling monclient init 2024-03-25T18:08:27.889 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.886+0000 7fa2d23ed640 10 monclient: init 2024-03-25T18:08:27.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.887+0000 7fa2d23ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:27.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.887+0000 7fa2d23ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:27.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.887+0000 7fa2d23ed640 10 monclient(hunting): picked mon.noname-a con 0x7fa2cc0b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:27.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.887+0000 7fa2d23ed640 10 monclient(hunting): picked mon.noname-c con 0x7fa2cc0b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:27.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.887+0000 7fa2d23ed640 10 monclient(hunting): picked mon.noname-b con 0x7fa2cc14be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:27.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.887+0000 7fa2d23ed640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2cc143220 con 0x7fa2cc0b1730 2024-03-25T18:08:27.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.887+0000 7fa2d23ed640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2cc1d7d30 con 0x7fa2cc14be90 2024-03-25T18:08:27.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.887+0000 7fa2d23ed640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2cc1379a0 con 0x7fa2cc0b4d60 2024-03-25T18:08:27.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.887+0000 7fa2d23ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:27.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.887+0000 7fa2d23ed640 10 monclient(hunting): authenticate will time out at 6863.139667s 2024-03-25T18:08:27.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.899+0000 7fa2cb7fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fa2cc0b4d60 0x7fa2cc1d7620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43090/0 (socket says 172.21.15.37:43090) 2024-03-25T18:08:27.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.899+0000 7fa2cb7fe640 1 -- 172.21.15.37:0/759246039 learned_addr learned my addr 172.21.15.37:0/759246039 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:27.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.903+0000 7fa2c8ff9640 10 client.?.objecter ms_handle_connect 0x7fa2cc14be90 2024-03-25T18:08:27.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.903+0000 7fa2c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:27.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.904+0000 7fa2c8ff9640 10 client.?.objecter ms_handle_connect 0x7fa2cc0b4d60 2024-03-25T18:08:27.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.904+0000 7fa2c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:27.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.908+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 398482265 0 0) 0x7fa2cc1d7d30 con 0x7fa2cc14be90 2024-03-25T18:08:27.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.908+0000 7fa2c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:27.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.908+0000 7fa2c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:27.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.908+0000 7fa2c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:27.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.908+0000 7fa2c8ff9640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:08:27.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.908+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2b8004eb0 con 0x7fa2cc14be90 2024-03-25T18:08:27.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.922+0000 7fa2c8ff9640 10 client.?.objecter ms_handle_connect 0x7fa2cc0b1730 2024-03-25T18:08:27.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.922+0000 7fa2c8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:27.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.922+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215735966 0 0) 0x7fa2cc1379a0 con 0x7fa2cc0b4d60 2024-03-25T18:08:27.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.922+0000 7fa2c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:27.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.922+0000 7fa2c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:27.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.922+0000 7fa2c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:27.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.922+0000 7fa2c8ff9640 10 monclient(hunting): my global_id is 4250 2024-03-25T18:08:27.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.922+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2cc1d7d30 con 0x7fa2cc0b4d60 2024-03-25T18:08:27.925 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.924+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270033531 0 0) 0x7fa2b8004eb0 con 0x7fa2cc14be90 2024-03-25T18:08:27.925 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.924+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2cc1379a0 con 0x7fa2cc14be90 2024-03-25T18:08:27.925 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.924+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa2c4001c10 con 0x7fa2cc14be90 2024-03-25T18:08:27.925 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.924+0000 7fa2c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.924+0000 7fa2c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.924+0000 7fa2c8ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:27.926 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.924+0000 7fa2c8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:27.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.936+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769028095 0 0) 0x7fa2cc143220 con 0x7fa2cc0b1730 2024-03-25T18:08:27.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.936+0000 7fa2c8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:27.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.936+0000 7fa2c8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:27.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.936+0000 7fa2c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:27.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.936+0000 7fa2c8ff9640 10 monclient(hunting): my global_id is 4248 2024-03-25T18:08:27.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.936+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2b8004eb0 con 0x7fa2cc0b1730 2024-03-25T18:08:27.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2027117655 0 0) 0x7fa2cc1379a0 con 0x7fa2cc14be90 2024-03-25T18:08:27.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 >> v1:172.21.15.37:6790/0 conn(0x7fa2cc0b4d60 legacy=0x7fa2cc1d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:27.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 >> v1:172.21.15.37:6789/0 conn(0x7fa2cc0b1730 legacy=0x7fa2cc142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:27.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2c8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:08:27.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2c8ff9640 1 monclient: found mon.b 2024-03-25T18:08:27.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2c8ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2cc140e20 con 0x7fa2cc14be90 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2c8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2c8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2d23ed640 5 monclient: authenticate success, global_id 4252 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2d23ed640 10 monclient: _renew_subs 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2d23ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2d23ed640 1 -- 172.21.15.37:0/759246039 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2cc138370 con 0x7fa2cc14be90 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2d23ed640 10 client.4252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2d23ed640 10 monclient: _renew_subs 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2d23ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2d23ed640 1 -- 172.21.15.37:0/759246039 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2cc140560 con 0x7fa2cc14be90 2024-03-25T18:08:27.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.945+0000 7fa2d23ed640 1 librados: init done 2024-03-25T18:08:27.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.946+0000 7fa2d23ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:27.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.946+0000 7fa2d23ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:27.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.946+0000 7fa2d23ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:27.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:27.946+0000 7fa2d23ed640 1 -- 172.21.15.37:0/759246039 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2cc143010 con 0x7fa2cc14be90 2024-03-25T18:08:28.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.226+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2c40020a0 con 0x7fa2cc14be90 2024-03-25T18:08:28.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.226+0000 7fa2c8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:28.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.226+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa2c4003b30 con 0x7fa2cc14be90 2024-03-25T18:08:28.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.226+0000 7fa2c8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:28.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.226+0000 7fa2c8ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:28.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.226+0000 7fa2c8ff9640 10 monclient: dump: 2024-03-25T18:08:28.228 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:28.228 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:28.228 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:28.228 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:28.228 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:28.228 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:28.228 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:28.228 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:28.228 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:28.228 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:28.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.226+0000 7fa2ca7fc640 4 set_mon_vals no callback set 2024-03-25T18:08:28.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.227+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fa2c4003d50 con 0x7fa2cc14be90 2024-03-25T18:08:28.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.228+0000 7fa2cbfff640 10 monclient: get_auth_request con 0x7fa2b806ddb0 auth_method 0 2024-03-25T18:08:28.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.228+0000 7fa2c8ff9640 10 client.4252.objecter ms_handle_connect 0x7fa2b806ddb0 2024-03-25T18:08:28.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.232+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fa2c40538b0 con 0x7fa2cc14be90 2024-03-25T18:08:28.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.232+0000 7fa2c8ff9640 10 client.4252.objecter ms_dispatch 0x7fa2cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:28.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.232+0000 7fa2c8ff9640 3 client.4252.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:28.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.232+0000 7fa2c8ff9640 3 client.4252.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:28.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.232+0000 7fa2c8ff9640 20 client.4252.objecter dump_active .. 0 homeless 2024-03-25T18:08:28.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.233+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2c404f1f0 con 0x7fa2cc14be90 2024-03-25T18:08:28.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.233+0000 7fa2c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:28.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.233+0000 7fa2c8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.493+0000 7fa2d23ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.493+0000 7fa2d23ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.493+0000 7fa2d23ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:28.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.493+0000 7fa2d23ed640 1 -- 172.21.15.37:0/759246039 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2cc0627b0 con 0x7fa2cc14be90 2024-03-25T18:08:28.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.523+0000 7fa2c8ff9640 1 -- 172.21.15.37:0/759246039 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fa2c40512f0 con 0x7fa2cc14be90 2024-03-25T18:08:28.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.523+0000 7fa2c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:28.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.523+0000 7fa2c8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:28.524 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:08:28.524 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.525+0000 7fa2d23ed640 10 librados: watch_flush enter 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.525+0000 7fa2d23ed640 10 librados: watch_flush exit 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.525+0000 7fa2d23ed640 20 client.4252.objecter shutdown clearing up homeless session... 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.525+0000 7fa2d23ed640 10 client.4252.objecter successfully canceled tick 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.525+0000 7fa2d23ed640 1 -- 172.21.15.37:0/759246039 >> v1:172.21.15.117:6800/3703993921 conn(0x7fa2b806ddb0 legacy=0x7fa2b8070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.525+0000 7fa2d23ed640 10 monclient: shutdown 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.525+0000 7fa2d23ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.525+0000 7fa2d23ed640 1 -- 172.21.15.37:0/759246039 >> v1:172.21.15.117:6789/0 conn(0x7fa2cc14be90 legacy=0x7fa2cc137290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.526+0000 7fa2d23ed640 1 -- 172.21.15.37:0/759246039 shutdown_connections 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.526+0000 7fa2d23ed640 1 -- 172.21.15.37:0/759246039 >> 172.21.15.37:0/759246039 conn(0x7fa2cc0617a0 msgr2=0x7fa2cc0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.526+0000 7fa2d23ed640 1 -- 172.21.15.37:0/759246039 shutdown_connections 2024-03-25T18:08:28.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.526+0000 7fa2d23ed640 1 -- 172.21.15.37:0/759246039 wait complete. 2024-03-25T18:08:28.528 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:28.526+0000 7fa2d23ed640 1 librados: shutdown 2024-03-25T18:08:28.539 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:35.346 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:35.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.460+0000 7f1b484ae640 10 monclient: get_monmap_and_config 2024-03-25T18:08:35.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.460+0000 7f1b484ae640 10 monclient: build_initial_monmap 2024-03-25T18:08:35.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.460+0000 7f1b484ae640 10 monclient: monmap: 2024-03-25T18:08:35.462 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:35.462 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:35.463 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:08:35.462673+0000 2024-03-25T18:08:35.463 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:08:35.462673+0000 2024-03-25T18:08:35.463 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:35.463 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:35.463 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:35.463 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:08:35.463 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:08:35.463 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.461+0000 7f1b484ae640 1 Processor -- start 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.461+0000 7f1b484ae640 1 -- start start 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.461+0000 7f1b484ae640 10 monclient: init 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.462+0000 7f1b484ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.462+0000 7f1b484ae640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.462+0000 7f1b484ae640 10 monclient(hunting): picked mon.noname-a con 0x7f1b400b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.462+0000 7f1b484ae640 10 monclient(hunting): picked mon.noname-c con 0x7f1b400b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.462+0000 7f1b484ae640 10 monclient(hunting): picked mon.noname-b con 0x7f1b4014be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.462+0000 7f1b484ae640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400af940 con 0x7f1b400b4d60 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.462+0000 7f1b484ae640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400b0a70 con 0x7f1b400b1730 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.462+0000 7f1b484ae640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b400b0d50 con 0x7f1b4014be90 2024-03-25T18:08:35.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.462+0000 7f1b484ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:35.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.462+0000 7f1b484ae640 10 monclient(hunting): authenticate will time out at 6870.715921s 2024-03-25T18:08:35.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.463+0000 7f1b45a49640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f1b400b1730 0x7f1b400b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:32876/0 (socket says 172.21.15.37:32876) 2024-03-25T18:08:35.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.463+0000 7f1b45a49640 1 -- 172.21.15.37:0/2890387713 learned_addr learned my addr 172.21.15.37:0/2890387713 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:35.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.550+0000 7f1b45248640 1 -- 172.21.15.37:0/2890387713 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055111127 0 0) 0x7f1b400af940 con 0x7f1b400b4d60 2024-03-25T18:08:35.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.550+0000 7f1b45248640 10 monclient(hunting): _negotiate 2024-03-25T18:08:35.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.550+0000 7f1b45248640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:35.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.550+0000 7f1b45248640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:35.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.550+0000 7f1b45248640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:08:35.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.550+0000 7f1b45248640 1 -- 172.21.15.37:0/2890387713 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b2c003520 con 0x7f1b400b4d60 2024-03-25T18:08:35.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 1 -- 172.21.15.37:0/2890387713 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2905286968 0 0) 0x7f1b2c003520 con 0x7f1b400b4d60 2024-03-25T18:08:35.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 1 -- 172.21.15.37:0/2890387713 >> v1:172.21.15.37:6790/0 conn(0x7f1b4014be90 legacy=0x7f1b4014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:35.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 1 -- 172.21.15.37:0/2890387713 >> v1:172.21.15.117:6789/0 conn(0x7f1b400b1730 legacy=0x7f1b400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:35.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 10 monclient: _finish_hunting 0 2024-03-25T18:08:35.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 1 monclient: found mon.noname-a 2024-03-25T18:08:35.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:35.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:08:35.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 1 -- 172.21.15.37:0/2890387713 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b400b1030 con 0x7f1b400b4d60 2024-03-25T18:08:35.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 10 monclient: _finish_auth 0 2024-03-25T18:08:35.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 10 monclient: _check_auth_tickets 2024-03-25T18:08:35.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:35.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 1 -- 172.21.15.37:0/2890387713 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1b3c001690 con 0x7f1b400b4d60 2024-03-25T18:08:35.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:35.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b45248640 10 monclient: dump: 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b484ae640 5 monclient: authenticate success, global_id 4251 2024-03-25T18:08:35.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.562+0000 7f1b484ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:35.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.579+0000 7f1b45248640 1 -- 172.21.15.37:0/2890387713 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b3c0017f0 con 0x7f1b400b4d60 2024-03-25T18:08:35.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.579+0000 7f1b45248640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:35.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.579+0000 7f1b45248640 1 -- 172.21.15.37:0/2890387713 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1b3c001c40 con 0x7f1b400b4d60 2024-03-25T18:08:35.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.579+0000 7f1b484ae640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.579+0000 7f1b484ae640 4 set_mon_vals no callback set 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.579+0000 7f1b45248640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.579+0000 7f1b45248640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.579+0000 7f1b45248640 10 monclient: dump: 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:35.581 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:35.582 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:35.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.579+0000 7f1b484ae640 10 monclient: shutdown 2024-03-25T18:08:35.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.579+0000 7f1b484ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:35.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.579+0000 7f1b484ae640 1 -- 172.21.15.37:0/2890387713 >> v1:172.21.15.37:6789/0 conn(0x7f1b400b4d60 legacy=0x7f1b4013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:35.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.580+0000 7f1b484ae640 1 -- 172.21.15.37:0/2890387713 shutdown_connections 2024-03-25T18:08:35.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.580+0000 7f1b484ae640 1 -- 172.21.15.37:0/2890387713 >> 172.21.15.37:0/2890387713 conn(0x7f1b400617a0 msgr2=0x7f1b40061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:35.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.580+0000 7f1b484ae640 1 -- 172.21.15.37:0/2890387713 shutdown_connections 2024-03-25T18:08:35.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.580+0000 7f1b484ae640 1 -- 172.21.15.37:0/2890387713 wait complete. 2024-03-25T18:08:35.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.580+0000 7f1b484ae640 10 monclient: build_initial_monmap 2024-03-25T18:08:35.584 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.580+0000 7f1b484ae640 10 monclient: monmap: 2024-03-25T18:08:35.584 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:35.584 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.581+0000 7f1b484ae640 1 librados: starting msgr at 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.581+0000 7f1b484ae640 1 librados: starting objecter 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.581+0000 7f1b484ae640 1 Processor -- start 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.581+0000 7f1b484ae640 1 -- start start 2024-03-25T18:08:35.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.581+0000 7f1b484ae640 1 librados: setting wanted keys 2024-03-25T18:08:35.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.581+0000 7f1b484ae640 1 librados: calling monclient init 2024-03-25T18:08:35.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.581+0000 7f1b484ae640 10 monclient: init 2024-03-25T18:08:35.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.581+0000 7f1b484ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:35.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.581+0000 7f1b484ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:08:35.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.581+0000 7f1b484ae640 10 monclient(hunting): picked mon.noname-c con 0x7f1b400b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:35.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.581+0000 7f1b484ae640 10 monclient(hunting): picked mon.noname-b con 0x7f1b400b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:35.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.582+0000 7f1b484ae640 10 monclient(hunting): picked mon.noname-a con 0x7f1b4014be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:35.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.582+0000 7f1b484ae640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b401368f0 con 0x7f1b4014be90 2024-03-25T18:08:35.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.582+0000 7f1b484ae640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b40137750 con 0x7f1b400b4d60 2024-03-25T18:08:35.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.582+0000 7f1b484ae640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b401363d0 con 0x7f1b400b1730 2024-03-25T18:08:35.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.582+0000 7f1b484ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:35.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.582+0000 7f1b484ae640 10 monclient(hunting): authenticate will time out at 6870.835377s 2024-03-25T18:08:35.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.585+0000 7f1b45a49640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f1b400b4d60 0x7f1b4012fd60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:32922/0 (socket says 172.21.15.37:32922) 2024-03-25T18:08:35.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.585+0000 7f1b45a49640 1 -- 172.21.15.37:0/1800174750 learned_addr learned my addr 172.21.15.37:0/1800174750 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:35.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.590+0000 7f1b36ffd640 10 client.?.objecter ms_handle_connect 0x7f1b400b4d60 2024-03-25T18:08:35.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.590+0000 7f1b36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:35.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.592+0000 7f1b36ffd640 10 client.?.objecter ms_handle_connect 0x7f1b4014be90 2024-03-25T18:08:35.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.592+0000 7f1b36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:35.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.602+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3858261966 0 0) 0x7f1b40137750 con 0x7f1b400b4d60 2024-03-25T18:08:35.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.602+0000 7f1b36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:35.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.602+0000 7f1b36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:35.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.602+0000 7f1b36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:35.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.602+0000 7f1b36ffd640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:08:35.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.602+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24004eb0 con 0x7f1b400b4d60 2024-03-25T18:08:35.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.612+0000 7f1b36ffd640 10 client.?.objecter ms_handle_connect 0x7f1b400b1730 2024-03-25T18:08:35.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.612+0000 7f1b36ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:35.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.612+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3104857240 0 0) 0x7f1b24004eb0 con 0x7f1b400b4d60 2024-03-25T18:08:35.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.612+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b40137750 con 0x7f1b400b4d60 2024-03-25T18:08:35.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.612+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1b300018b0 con 0x7f1b400b4d60 2024-03-25T18:08:35.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.612+0000 7f1b36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:35.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.612+0000 7f1b36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:35.614 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.612+0000 7f1b36ffd640 10 monclient(hunting): dump: 2024-03-25T18:08:35.615 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:35.615 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:35.615 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:35.615 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:35.615 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:35.615 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:35.615 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:35.615 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:35.615 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:35.615 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:35.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.612+0000 7f1b36ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:35.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.614+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888089957 0 0) 0x7f1b401368f0 con 0x7f1b4014be90 2024-03-25T18:08:35.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.614+0000 7f1b36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:35.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.614+0000 7f1b36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:35.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.614+0000 7f1b36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:35.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.614+0000 7f1b36ffd640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:08:35.616 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.614+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b24004eb0 con 0x7f1b4014be90 2024-03-25T18:08:35.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.623+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3594381071 0 0) 0x7f1b401363d0 con 0x7f1b400b1730 2024-03-25T18:08:35.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.623+0000 7f1b36ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:35.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.623+0000 7f1b36ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:35.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.623+0000 7f1b36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:35.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.623+0000 7f1b36ffd640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:08:35.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.623+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b401368f0 con 0x7f1b400b1730 2024-03-25T18:08:35.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3197849118 0 0) 0x7f1b40137750 con 0x7f1b400b4d60 2024-03-25T18:08:35.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 >> v1:172.21.15.37:6790/0 conn(0x7f1b400b1730 legacy=0x7f1b40135b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:35.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 >> v1:172.21.15.37:6789/0 conn(0x7f1b4014be90 legacy=0x7f1b401d7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:35.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b36ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:35.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b36ffd640 1 monclient: found mon.b 2024-03-25T18:08:35.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:35.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b36ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:35.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b40130470 con 0x7f1b400b4d60 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b36ffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b36ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b484ae640 5 monclient: authenticate success, global_id 4258 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b484ae640 10 monclient: _renew_subs 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b484ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b484ae640 1 -- 172.21.15.37:0/1800174750 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b40130720 con 0x7f1b400b4d60 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b484ae640 10 client.4258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b484ae640 10 monclient: _renew_subs 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b484ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b484ae640 1 -- 172.21.15.37:0/1800174750 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b401322f0 con 0x7f1b400b4d60 2024-03-25T18:08:35.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.630+0000 7f1b484ae640 1 librados: init done 2024-03-25T18:08:35.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.631+0000 7f1b484ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:35.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.631+0000 7f1b484ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:35.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.631+0000 7f1b484ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:35.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.631+0000 7f1b484ae640 1 -- 172.21.15.37:0/1800174750 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b40136210 con 0x7f1b400b4d60 2024-03-25T18:08:35.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.642+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b300047e0 con 0x7f1b400b4d60 2024-03-25T18:08:35.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.642+0000 7f1b36ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:35.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.642+0000 7f1b44a47640 4 set_mon_vals no callback set 2024-03-25T18:08:35.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.935+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1b30004c30 con 0x7f1b400b4d60 2024-03-25T18:08:35.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.935+0000 7f1b36ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:35.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.935+0000 7f1b36ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:35.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.935+0000 7f1b36ffd640 10 monclient: dump: 2024-03-25T18:08:35.937 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:35.937 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:35.937 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:35.937 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:35.937 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:35.937 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:35.938 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:35.938 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:35.938 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:35.938 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:35.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.936+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f1b30004e10 con 0x7f1b400b4d60 2024-03-25T18:08:35.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.937+0000 7f1b4624a640 10 monclient: get_auth_request con 0x7f1b2406de80 auth_method 0 2024-03-25T18:08:35.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.937+0000 7f1b36ffd640 10 client.4258.objecter ms_handle_connect 0x7f1b2406de80 2024-03-25T18:08:35.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.939+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f1b300548a0 con 0x7f1b400b4d60 2024-03-25T18:08:35.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.939+0000 7f1b36ffd640 10 client.4258.objecter ms_dispatch 0x7f1b40000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:35.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.939+0000 7f1b36ffd640 3 client.4258.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:35.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.939+0000 7f1b36ffd640 3 client.4258.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:35.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.939+0000 7f1b36ffd640 20 client.4258.objecter dump_active .. 0 homeless 2024-03-25T18:08:35.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.941+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b300503b0 con 0x7f1b400b4d60 2024-03-25T18:08:35.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.941+0000 7f1b36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:35.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:35.941+0000 7f1b36ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:36.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.189+0000 7f1b484ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:36.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.189+0000 7f1b484ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:36.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.189+0000 7f1b484ae640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:36.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.189+0000 7f1b484ae640 1 -- 172.21.15.37:0/1800174750 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b40136560 con 0x7f1b400b4d60 2024-03-25T18:08:36.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.231+0000 7f1b36ffd640 1 -- 172.21.15.37:0/1800174750 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f1b300522e0 con 0x7f1b400b4d60 2024-03-25T18:08:36.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.231+0000 7f1b36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:36.233 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.231+0000 7f1b36ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:36.233 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:08:36.233 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.234+0000 7f1b484ae640 10 librados: watch_flush enter 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.234+0000 7f1b484ae640 10 librados: watch_flush exit 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.234+0000 7f1b484ae640 20 client.4258.objecter shutdown clearing up homeless session... 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.234+0000 7f1b484ae640 10 client.4258.objecter successfully canceled tick 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.234+0000 7f1b484ae640 1 -- 172.21.15.37:0/1800174750 >> v1:172.21.15.117:6800/3703993921 conn(0x7f1b2406de80 legacy=0x7f1b24070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.234+0000 7f1b484ae640 10 monclient: shutdown 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.234+0000 7f1b484ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.234+0000 7f1b484ae640 1 -- 172.21.15.37:0/1800174750 >> v1:172.21.15.117:6789/0 conn(0x7f1b400b4d60 legacy=0x7f1b4012fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.234+0000 7f1b484ae640 1 -- 172.21.15.37:0/1800174750 shutdown_connections 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.235+0000 7f1b484ae640 1 -- 172.21.15.37:0/1800174750 >> 172.21.15.37:0/1800174750 conn(0x7f1b400617a0 msgr2=0x7f1b40138500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.235+0000 7f1b484ae640 1 -- 172.21.15.37:0/1800174750 shutdown_connections 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.235+0000 7f1b484ae640 1 -- 172.21.15.37:0/1800174750 wait complete. 2024-03-25T18:08:36.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:36.235+0000 7f1b484ae640 1 librados: shutdown 2024-03-25T18:08:36.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:39.132 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:01:14.477 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:39.158 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:01:14.472 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:39.195 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:01:14.541 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:39.271 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:01:14.584 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:39.351 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:01:14.697 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:39.415 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:01:14.761 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:39.431 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:01:14.777 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:39.499 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:01:14.813 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:39.661 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:01:15.007 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:43.055 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:43.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.169+0000 7fe56e4c8640 10 monclient: get_monmap_and_config 2024-03-25T18:08:43.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.169+0000 7fe56e4c8640 10 monclient: build_initial_monmap 2024-03-25T18:08:43.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.169+0000 7fe56e4c8640 10 monclient: monmap: 2024-03-25T18:08:43.170 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:43.170 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:43.170 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:08:43.170281+0000 2024-03-25T18:08:43.170 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:08:43.170281+0000 2024-03-25T18:08:43.170 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:43.171 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:43.171 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:43.171 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:08:43.171 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:08:43.171 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:43.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 1 Processor -- start 2024-03-25T18:08:43.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 1 -- start start 2024-03-25T18:08:43.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 10 monclient: init 2024-03-25T18:08:43.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:43.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:08:43.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe5680a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:43.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe56813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:43.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe5681430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:43.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe568142520 con 0x7fe5680a98f0 2024-03-25T18:08:43.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5680b28a0 con 0x7fe5681430c0 2024-03-25T18:08:43.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5680b2a30 con 0x7fe56813f710 2024-03-25T18:08:43.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.170+0000 7fe56e4c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:43.173 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.171+0000 7fe56e4c8640 10 monclient(hunting): authenticate will time out at 6878.423292s 2024-03-25T18:08:43.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.177+0000 7fe567fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe5680a98f0 0x7fe5680a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:33474/0 (socket says 172.21.15.37:33474) 2024-03-25T18:08:43.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.177+0000 7fe567fff640 1 -- 172.21.15.37:0/3473414183 learned_addr learned my addr 172.21.15.37:0/3473414183 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:43.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.189+0000 7fe566ffd640 1 -- 172.21.15.37:0/3473414183 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 347017575 0 0) 0x7fe5680b2a30 con 0x7fe56813f710 2024-03-25T18:08:43.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.189+0000 7fe566ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.189+0000 7fe566ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.189+0000 7fe566ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.189+0000 7fe566ffd640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:08:43.191 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.189+0000 7fe566ffd640 1 -- 172.21.15.37:0/3473414183 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe550003520 con 0x7fe56813f710 2024-03-25T18:08:43.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 1 -- 172.21.15.37:0/3473414183 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1922634440 0 0) 0x7fe550003520 con 0x7fe56813f710 2024-03-25T18:08:43.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 1 -- 172.21.15.37:0/3473414183 >> v1:172.21.15.117:6789/0 conn(0x7fe5681430c0 legacy=0x7fe56814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 1 -- 172.21.15.37:0/3473414183 >> v1:172.21.15.37:6789/0 conn(0x7fe5680a98f0 legacy=0x7fe5680a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 1 monclient: found mon.noname-b 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 1 -- 172.21.15.37:0/3473414183 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5680b44c0 con 0x7fe56813f710 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 1 -- 172.21.15.37:0/3473414183 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe55c001690 con 0x7fe56813f710 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe566ffd640 10 monclient: dump: 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.197 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.198 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.198 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:43.198 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:43.198 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:43.198 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:43.198 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:43.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe56e4c8640 5 monclient: authenticate success, global_id 4259 2024-03-25T18:08:43.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.195+0000 7fe56e4c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:43.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.205+0000 7fe566ffd640 1 -- 172.21.15.37:0/3473414183 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe55c0017f0 con 0x7fe56813f710 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.205+0000 7fe566ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.205+0000 7fe566ffd640 1 -- 172.21.15.37:0/3473414183 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe55c001c40 con 0x7fe56813f710 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.205+0000 7fe56e4c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.205+0000 7fe56e4c8640 4 set_mon_vals no callback set 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.205+0000 7fe566ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.205+0000 7fe566ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.205+0000 7fe566ffd640 10 monclient: dump: 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:43.207 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:43.208 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:43.208 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:43.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.206+0000 7fe56e4c8640 10 monclient: shutdown 2024-03-25T18:08:43.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.206+0000 7fe56e4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:43.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.206+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3473414183 >> v1:172.21.15.37:6790/0 conn(0x7fe56813f710 legacy=0x7fe5681429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.206+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3473414183 shutdown_connections 2024-03-25T18:08:43.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.206+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3473414183 >> 172.21.15.37:0/3473414183 conn(0x7fe5680b7ea0 msgr2=0x7fe5680b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:43.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.206+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3473414183 shutdown_connections 2024-03-25T18:08:43.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.206+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3473414183 wait complete. 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.207+0000 7fe56e4c8640 10 monclient: build_initial_monmap 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.207+0000 7fe56e4c8640 10 monclient: monmap: 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.208+0000 7fe56e4c8640 1 librados: starting msgr at 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.208+0000 7fe56e4c8640 1 librados: starting objecter 2024-03-25T18:08:43.210 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.208+0000 7fe56e4c8640 1 Processor -- start 2024-03-25T18:08:43.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.208+0000 7fe56e4c8640 1 -- start start 2024-03-25T18:08:43.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.208+0000 7fe56e4c8640 1 librados: setting wanted keys 2024-03-25T18:08:43.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.208+0000 7fe56e4c8640 1 librados: calling monclient init 2024-03-25T18:08:43.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.208+0000 7fe56e4c8640 10 monclient: init 2024-03-25T18:08:43.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.209+0000 7fe56e4c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:43.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.209+0000 7fe56e4c8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:08:43.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.209+0000 7fe56e4c8640 10 monclient(hunting): picked mon.noname-c con 0x7fe5680a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:43.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.209+0000 7fe56e4c8640 10 monclient(hunting): picked mon.noname-a con 0x7fe56813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:43.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.209+0000 7fe56e4c8640 10 monclient(hunting): picked mon.noname-b con 0x7fe5681430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:43.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.209+0000 7fe56e4c8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe568138c10 con 0x7fe56813f710 2024-03-25T18:08:43.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.209+0000 7fe56e4c8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe568138e60 con 0x7fe5681430c0 2024-03-25T18:08:43.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.209+0000 7fe56e4c8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe568139140 con 0x7fe5680a98f0 2024-03-25T18:08:43.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.209+0000 7fe56e4c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:43.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.209+0000 7fe56e4c8640 10 monclient(hunting): authenticate will time out at 6878.461910s 2024-03-25T18:08:43.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.221+0000 7fe567fff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fe5680a98f0 0x7fe568137bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:40998/0 (socket says 172.21.15.37:40998) 2024-03-25T18:08:43.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.221+0000 7fe567fff640 1 -- 172.21.15.37:0/3282285461 learned_addr learned my addr 172.21.15.37:0/3282285461 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.224+0000 7fe564ff9640 10 client.?.objecter ms_handle_connect 0x7fe56813f710 2024-03-25T18:08:43.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.224+0000 7fe564ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:43.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.226+0000 7fe564ff9640 10 client.?.objecter ms_handle_connect 0x7fe5680a98f0 2024-03-25T18:08:43.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.226+0000 7fe564ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:43.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.228+0000 7fe564ff9640 10 client.?.objecter ms_handle_connect 0x7fe5681430c0 2024-03-25T18:08:43.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.228+0000 7fe564ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:43.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.236+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249185476 0 0) 0x7fe568138c10 con 0x7fe56813f710 2024-03-25T18:08:43.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.236+0000 7fe564ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.236+0000 7fe564ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.236+0000 7fe564ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.236+0000 7fe564ff9640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:08:43.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.236+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe554004eb0 con 0x7fe56813f710 2024-03-25T18:08:43.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.237+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 606791422 0 0) 0x7fe568139140 con 0x7fe5680a98f0 2024-03-25T18:08:43.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.237+0000 7fe564ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.237+0000 7fe564ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.237+0000 7fe564ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.237+0000 7fe564ff9640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:08:43.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.237+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe568138c10 con 0x7fe5680a98f0 2024-03-25T18:08:43.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.241+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288230209 0 0) 0x7fe568138e60 con 0x7fe5681430c0 2024-03-25T18:08:43.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.241+0000 7fe564ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:43.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.241+0000 7fe564ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:43.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.241+0000 7fe564ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:43.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.241+0000 7fe564ff9640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:08:43.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.242+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe568139140 con 0x7fe5681430c0 2024-03-25T18:08:43.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.244+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 635405443 0 0) 0x7fe568138c10 con 0x7fe5680a98f0 2024-03-25T18:08:43.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.244+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe568138e60 con 0x7fe5680a98f0 2024-03-25T18:08:43.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.244+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe5580018b0 con 0x7fe5680a98f0 2024-03-25T18:08:43.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.244+0000 7fe564ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.244+0000 7fe564ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.244+0000 7fe564ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:43.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.244+0000 7fe564ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:43.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.245+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1115048428 0 0) 0x7fe554004eb0 con 0x7fe56813f710 2024-03-25T18:08:43.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.245+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe568138c10 con 0x7fe56813f710 2024-03-25T18:08:43.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.245+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe55c001bc0 con 0x7fe56813f710 2024-03-25T18:08:43.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.245+0000 7fe564ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.245+0000 7fe564ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:43.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.245+0000 7fe564ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:43.247 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:43.248 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:43.248 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.248 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.248 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.248 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:43.248 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:43.248 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:43.248 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:43.248 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.261+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2432174898 0 0) 0x7fe568139140 con 0x7fe5681430c0 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.261+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe554004eb0 con 0x7fe5681430c0 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.261+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe560001a60 con 0x7fe5681430c0 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.261+0000 7fe564ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.261+0000 7fe564ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.261+0000 7fe564ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:43.262 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:43.263 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:43.263 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:43.263 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1119804678 0 0) 0x7fe568138c10 con 0x7fe56813f710 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 >> v1:172.21.15.37:6790/0 conn(0x7fe5680a98f0 legacy=0x7fe568137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 >> v1:172.21.15.117:6789/0 conn(0x7fe5681430c0 legacy=0x7fe5681e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe564ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe564ff9640 1 monclient: found mon.a 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe564ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe564ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe568139920 con 0x7fe56813f710 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe564ff9640 10 monclient: _finish_auth 0 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe564ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe564ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:43.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe56e4c8640 5 monclient: authenticate success, global_id 4260 2024-03-25T18:08:43.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe56e4c8640 10 monclient: _renew_subs 2024-03-25T18:08:43.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe56e4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:08:43.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3282285461 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5681382e0 con 0x7fe56813f710 2024-03-25T18:08:43.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe56e4c8640 10 client.4260.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:43.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe56e4c8640 10 monclient: _renew_subs 2024-03-25T18:08:43.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe56e4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:08:43.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3282285461 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5680ab8b0 con 0x7fe56813f710 2024-03-25T18:08:43.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.263+0000 7fe56e4c8640 1 librados: init done 2024-03-25T18:08:43.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.264+0000 7fe56e4c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:43.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.264+0000 7fe56e4c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:43.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.264+0000 7fe56e4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:08:43.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.264+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3282285461 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe568137a20 con 0x7fe56813f710 2024-03-25T18:08:43.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.574+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe55c003960 con 0x7fe56813f710 2024-03-25T18:08:43.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.574+0000 7fe564ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:43.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.574+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe55c003c70 con 0x7fe56813f710 2024-03-25T18:08:43.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.574+0000 7fe564ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:43.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.574+0000 7fe564ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:08:43.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.574+0000 7fe564ff9640 10 monclient: dump: 2024-03-25T18:08:43.575 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:43.575 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:43.575 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.575 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:43.575 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:43.576 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:43.576 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:43.576 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:43.576 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:43.576 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:43.576 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.574+0000 7fe5667fc640 4 set_mon_vals no callback set 2024-03-25T18:08:43.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.575+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fe55c019c40 con 0x7fe56813f710 2024-03-25T18:08:43.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.575+0000 7fe567fff640 10 monclient: get_auth_request con 0x7fe55406e110 auth_method 0 2024-03-25T18:08:43.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.575+0000 7fe564ff9640 10 client.4260.objecter ms_handle_connect 0x7fe55406e110 2024-03-25T18:08:43.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.587+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fe55c053730 con 0x7fe56813f710 2024-03-25T18:08:43.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.587+0000 7fe564ff9640 10 client.4260.objecter ms_dispatch 0x7fe568000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:43.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.587+0000 7fe564ff9640 3 client.4260.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:43.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.587+0000 7fe564ff9640 3 client.4260.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:43.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.587+0000 7fe564ff9640 20 client.4260.objecter dump_active .. 0 homeless 2024-03-25T18:08:43.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.587+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe55c04f240 con 0x7fe56813f710 2024-03-25T18:08:43.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.587+0000 7fe564ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:43.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.587+0000 7fe564ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:43.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.811+0000 7fe56e4c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:43.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.811+0000 7fe56e4c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:43.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.811+0000 7fe56e4c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:08:43.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.811+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3282285461 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe56813d010 con 0x7fe56813f710 2024-03-25T18:08:43.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.849+0000 7fe564ff9640 1 -- 172.21.15.37:0/3282285461 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fe55c051170 con 0x7fe56813f710 2024-03-25T18:08:43.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.849+0000 7fe564ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:43.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.849+0000 7fe564ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:43.850 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:08:43.850 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:43.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.851+0000 7fe56e4c8640 10 librados: watch_flush enter 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.851+0000 7fe56e4c8640 10 librados: watch_flush exit 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.851+0000 7fe56e4c8640 20 client.4260.objecter shutdown clearing up homeless session... 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.851+0000 7fe56e4c8640 10 client.4260.objecter successfully canceled tick 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.851+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3282285461 >> v1:172.21.15.117:6800/3703993921 conn(0x7fe55406e110 legacy=0x7fe5540705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.851+0000 7fe56e4c8640 10 monclient: shutdown 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.851+0000 7fe56e4c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.851+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3282285461 >> v1:172.21.15.37:6789/0 conn(0x7fe56813f710 legacy=0x7fe568132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.851+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3282285461 shutdown_connections 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.852+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3282285461 >> 172.21.15.37:0/3282285461 conn(0x7fe5680b7ea0 msgr2=0x7fe568074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.852+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3282285461 shutdown_connections 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.852+0000 7fe56e4c8640 1 -- 172.21.15.37:0/3282285461 wait complete. 2024-03-25T18:08:43.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:43.852+0000 7fe56e4c8640 1 librados: shutdown 2024-03-25T18:08:43.865 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:50.672 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:50.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.791+0000 7f7e7a47b640 10 monclient: get_monmap_and_config 2024-03-25T18:08:50.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.791+0000 7f7e7a47b640 10 monclient: build_initial_monmap 2024-03-25T18:08:50.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.791+0000 7f7e7a47b640 10 monclient: monmap: 2024-03-25T18:08:50.792 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:50.792 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:50.792 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:08:50.792166+0000 2024-03-25T18:08:50.792 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:08:50.792166+0000 2024-03-25T18:08:50.792 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:50.793 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:50.793 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:50.793 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:08:50.793 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:08:50.793 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:50.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.792+0000 7f7e7a47b640 1 Processor -- start 2024-03-25T18:08:50.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.792+0000 7f7e7a47b640 1 -- start start 2024-03-25T18:08:50.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.792+0000 7f7e7a47b640 10 monclient: init 2024-03-25T18:08:50.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.793+0000 7f7e7a47b640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:50.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.793+0000 7f7e7a47b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:50.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.793+0000 7f7e7a47b640 10 monclient(hunting): picked mon.noname-c con 0x7f7e740a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:50.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.793+0000 7f7e7a47b640 10 monclient(hunting): picked mon.noname-a con 0x7f7e7413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:50.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.793+0000 7f7e7a47b640 10 monclient(hunting): picked mon.noname-b con 0x7f7e741430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:50.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.793+0000 7f7e7a47b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e74142520 con 0x7f7e7413f710 2024-03-25T18:08:50.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.793+0000 7f7e7a47b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e740b28a0 con 0x7f7e740a98f0 2024-03-25T18:08:50.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.793+0000 7f7e7a47b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e740b2a30 con 0x7f7e741430c0 2024-03-25T18:08:50.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.793+0000 7f7e7a47b640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:50.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.793+0000 7f7e7a47b640 10 monclient(hunting): authenticate will time out at 6886.045622s 2024-03-25T18:08:50.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.808+0000 7f7e78a18640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f7e741430c0 0x7f7e7414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:58546/0 (socket says 172.21.15.37:58546) 2024-03-25T18:08:50.809 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.808+0000 7f7e78a18640 1 -- 172.21.15.37:0/1646833896 learned_addr learned my addr 172.21.15.37:0/1646833896 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:50.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.815+0000 7f7e72ffd640 1 -- 172.21.15.37:0/1646833896 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 233762107 0 0) 0x7f7e740b2a30 con 0x7f7e741430c0 2024-03-25T18:08:50.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.815+0000 7f7e72ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:50.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.815+0000 7f7e72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:50.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.815+0000 7f7e72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:50.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.815+0000 7f7e72ffd640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:08:50.817 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.815+0000 7f7e72ffd640 1 -- 172.21.15.37:0/1646833896 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e5c003520 con 0x7f7e741430c0 2024-03-25T18:08:50.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 1 -- 172.21.15.37:0/1646833896 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2099269755 0 0) 0x7f7e5c003520 con 0x7f7e741430c0 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 1 -- 172.21.15.37:0/1646833896 >> v1:172.21.15.117:6789/0 conn(0x7f7e740a98f0 legacy=0x7f7e740a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 1 -- 172.21.15.37:0/1646833896 >> v1:172.21.15.37:6789/0 conn(0x7f7e7413f710 legacy=0x7f7e741429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 1 monclient: found mon.noname-b 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 1 -- 172.21.15.37:0/1646833896 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e740b44c0 con 0x7f7e741430c0 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 1 -- 172.21.15.37:0/1646833896 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7e6c001a10 con 0x7f7e741430c0 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e72ffd640 10 monclient: dump: 2024-03-25T18:08:50.826 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:50.827 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:50.827 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:50.827 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:50.827 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:50.827 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:50.827 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:50.827 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:50.827 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:50.827 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:50.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e7a47b640 5 monclient: authenticate success, global_id 4265 2024-03-25T18:08:50.827 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.824+0000 7f7e7a47b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:50.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.831+0000 7f7e72ffd640 1 -- 172.21.15.37:0/1646833896 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e6c001b70 con 0x7f7e741430c0 2024-03-25T18:08:50.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.831+0000 7f7e72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:50.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.831+0000 7f7e72ffd640 1 -- 172.21.15.37:0/1646833896 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7e6c001fc0 con 0x7f7e741430c0 2024-03-25T18:08:50.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.831+0000 7f7e7a47b640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:50.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.831+0000 7f7e7a47b640 4 set_mon_vals no callback set 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.831+0000 7f7e72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.831+0000 7f7e72ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.831+0000 7f7e72ffd640 10 monclient: dump: 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:50.833 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:50.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.832+0000 7f7e7a47b640 10 monclient: shutdown 2024-03-25T18:08:50.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.832+0000 7f7e7a47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:50.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.832+0000 7f7e7a47b640 1 -- 172.21.15.37:0/1646833896 >> v1:172.21.15.37:6790/0 conn(0x7f7e741430c0 legacy=0x7f7e7414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:50.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.832+0000 7f7e7a47b640 1 -- 172.21.15.37:0/1646833896 shutdown_connections 2024-03-25T18:08:50.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.832+0000 7f7e7a47b640 1 -- 172.21.15.37:0/1646833896 >> 172.21.15.37:0/1646833896 conn(0x7f7e740b7ea0 msgr2=0x7f7e740b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:50.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.833+0000 7f7e7a47b640 1 -- 172.21.15.37:0/1646833896 shutdown_connections 2024-03-25T18:08:50.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.833+0000 7f7e7a47b640 1 -- 172.21.15.37:0/1646833896 wait complete. 2024-03-25T18:08:50.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.833+0000 7f7e7a47b640 10 monclient: build_initial_monmap 2024-03-25T18:08:50.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.833+0000 7f7e7a47b640 10 monclient: monmap: 2024-03-25T18:08:50.835 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:50.835 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.834+0000 7f7e7a47b640 1 librados: starting msgr at 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.834+0000 7f7e7a47b640 1 librados: starting objecter 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.834+0000 7f7e7a47b640 1 Processor -- start 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.834+0000 7f7e7a47b640 1 -- start start 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.834+0000 7f7e7a47b640 1 librados: setting wanted keys 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.834+0000 7f7e7a47b640 1 librados: calling monclient init 2024-03-25T18:08:50.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.834+0000 7f7e7a47b640 10 monclient: init 2024-03-25T18:08:50.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.835+0000 7f7e7a47b640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:50.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.835+0000 7f7e7a47b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:08:50.837 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.835+0000 7f7e7a47b640 10 monclient(hunting): picked mon.noname-b con 0x7f7e740a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:50.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.835+0000 7f7e7a47b640 10 monclient(hunting): picked mon.noname-a con 0x7f7e7413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:50.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.835+0000 7f7e7a47b640 10 monclient(hunting): picked mon.noname-c con 0x7f7e741430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:50.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.835+0000 7f7e7a47b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e74136ec0 con 0x7f7e7413f710 2024-03-25T18:08:50.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.835+0000 7f7e7a47b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e74136c20 con 0x7f7e740a98f0 2024-03-25T18:08:50.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.835+0000 7f7e7a47b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e74137db0 con 0x7f7e741430c0 2024-03-25T18:08:50.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.835+0000 7f7e7a47b640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:50.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.835+0000 7f7e7a47b640 10 monclient(hunting): authenticate will time out at 6886.087571s 2024-03-25T18:08:50.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.847+0000 7f7e73fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7e740a98f0 0x7f7e741376a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:33042/0 (socket says 172.21.15.37:33042) 2024-03-25T18:08:50.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.847+0000 7f7e73fff640 1 -- 172.21.15.37:0/166582887 learned_addr learned my addr 172.21.15.37:0/166582887 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:50.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.849+0000 7f7e70ff9640 10 client.?.objecter ms_handle_connect 0x7f7e740a98f0 2024-03-25T18:08:50.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.849+0000 7f7e70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:50.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.854+0000 7f7e70ff9640 10 client.?.objecter ms_handle_connect 0x7f7e7413f710 2024-03-25T18:08:50.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.854+0000 7f7e70ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:50.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.860+0000 7f7e78a18640 1 -- 172.21.15.37:0/166582887 >> v1:172.21.15.37:6790/0 conn(0x7f7e741430c0 legacy=0x7f7e741e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:08:50.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.860+0000 7f7e78a18640 1 -- 172.21.15.37:0/166582887 >> v1:172.21.15.37:6790/0 conn(0x7f7e741430c0 legacy=0x7f7e741e2c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:08:50.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.860+0000 7f7e78a18640 1 --1- 172.21.15.37:0/166582887 >> v1:172.21.15.37:6790/0 conn(0x7f7e741430c0 0x7f7e741e2c30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).handle_connect_reply_1 read connect reply failed 2024-03-25T18:08:50.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.860+0000 7f7e78a18640 1 --1- 172.21.15.37:0/166582887 >> v1:172.21.15.37:6790/0 conn(0x7f7e741430c0 0x7f7e741e2c30 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=1).fault on lossy channel, failing 2024-03-25T18:08:50.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.860+0000 7f7e70ff9640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.37:6790/0 2024-03-25T18:08:50.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.866+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166367185 0 0) 0x7f7e74136c20 con 0x7f7e740a98f0 2024-03-25T18:08:50.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.866+0000 7f7e70ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:50.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.866+0000 7f7e70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:50.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.866+0000 7f7e70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:50.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.866+0000 7f7e70ff9640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:08:50.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.866+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e60004eb0 con 0x7f7e740a98f0 2024-03-25T18:08:50.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.871+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 763987779 0 0) 0x7f7e74136ec0 con 0x7f7e7413f710 2024-03-25T18:08:50.872 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.871+0000 7f7e70ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:50.872 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.871+0000 7f7e70ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:50.872 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.871+0000 7f7e70ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:50.872 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.871+0000 7f7e70ff9640 10 monclient(hunting): my global_id is 4266 2024-03-25T18:08:50.872 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.871+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e74136c20 con 0x7f7e7413f710 2024-03-25T18:08:50.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.889+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3991191035 0 0) 0x7f7e74136c20 con 0x7f7e7413f710 2024-03-25T18:08:50.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.889+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e74136ec0 con 0x7f7e7413f710 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.889+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7e680018b0 con 0x7f7e7413f710 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.889+0000 7f7e70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.889+0000 7f7e70ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.889+0000 7f7e70ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:50.891 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:50.892 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:50.892 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:50.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.889+0000 7f7e70ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:50.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.896+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1826825401 0 0) 0x7f7e60004eb0 con 0x7f7e740a98f0 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.896+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7e74136c20 con 0x7f7e740a98f0 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.896+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7e640019d0 con 0x7f7e740a98f0 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.896+0000 7f7e70ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.896+0000 7f7e70ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.896+0000 7f7e70ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:50.897 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:50.898 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:50.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1402976112 0 0) 0x7f7e74136c20 con 0x7f7e740a98f0 2024-03-25T18:08:50.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 >> v1:172.21.15.37:6790/0 conn(0x7f7e741430c0 legacy=0x7f7e741e2c30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:08:50.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 >> v1:172.21.15.37:6789/0 conn(0x7f7e7413f710 legacy=0x7f7e740b0d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:50.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e70ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:08:50.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e70ff9640 1 monclient: found mon.b 2024-03-25T18:08:50.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e70ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:50.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e70ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e74138b50 con 0x7f7e740a98f0 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e70ff9640 10 monclient: _finish_auth 0 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e70ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e70ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e7a47b640 5 monclient: authenticate success, global_id 4270 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e7a47b640 10 monclient: _renew_subs 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e7a47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e7a47b640 1 -- 172.21.15.37:0/166582887 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e74139830 con 0x7f7e740a98f0 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e7a47b640 10 client.4270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e7a47b640 10 monclient: _renew_subs 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e7a47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e7a47b640 1 -- 172.21.15.37:0/166582887 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e741369b0 con 0x7f7e740a98f0 2024-03-25T18:08:50.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.904+0000 7f7e7a47b640 1 librados: init done 2024-03-25T18:08:50.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.905+0000 7f7e7a47b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:50.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.905+0000 7f7e7a47b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:50.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.905+0000 7f7e7a47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:50.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:50.905+0000 7f7e7a47b640 1 -- 172.21.15.37:0/166582887 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e7414f320 con 0x7f7e740a98f0 2024-03-25T18:08:51.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.180+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7e64003610 con 0x7f7e740a98f0 2024-03-25T18:08:51.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.180+0000 7f7e70ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:51.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.180+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7e64003960 con 0x7f7e740a98f0 2024-03-25T18:08:51.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.180+0000 7f7e70ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:51.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.180+0000 7f7e70ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:51.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.180+0000 7f7e70ff9640 10 monclient: dump: 2024-03-25T18:08:51.181 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:51.182 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:51.182 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:51.182 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:51.182 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:51.182 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:51.182 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:51.182 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:51.182 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:51.182 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:51.182 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.180+0000 7f7e727fc640 4 set_mon_vals no callback set 2024-03-25T18:08:51.182 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.180+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f7e64003b80 con 0x7f7e740a98f0 2024-03-25T18:08:51.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.181+0000 7f7e737fe640 10 monclient: get_auth_request con 0x7f7e6006de90 auth_method 0 2024-03-25T18:08:51.183 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.181+0000 7f7e70ff9640 10 client.4270.objecter ms_handle_connect 0x7f7e6006de90 2024-03-25T18:08:51.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.186+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f7e640547a0 con 0x7f7e740a98f0 2024-03-25T18:08:51.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.186+0000 7f7e70ff9640 10 client.4270.objecter ms_dispatch 0x7f7e74000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:51.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.186+0000 7f7e70ff9640 3 client.4270.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:51.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.186+0000 7f7e70ff9640 3 client.4270.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:51.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.186+0000 7f7e70ff9640 20 client.4270.objecter dump_active .. 0 homeless 2024-03-25T18:08:51.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.188+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7e640502b0 con 0x7f7e740a98f0 2024-03-25T18:08:51.188 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.188+0000 7f7e70ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:51.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.188+0000 7f7e70ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:51.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.409+0000 7f7e7a47b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:51.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.409+0000 7f7e7a47b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:51.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.409+0000 7f7e7a47b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:51.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.409+0000 7f7e7a47b640 1 -- 172.21.15.37:0/166582887 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7e740627b0 con 0x7f7e740a98f0 2024-03-25T18:08:51.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.440+0000 7f7e70ff9640 1 -- 172.21.15.37:0/166582887 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f7e640521e0 con 0x7f7e740a98f0 2024-03-25T18:08:51.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.440+0000 7f7e70ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:51.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.440+0000 7f7e70ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:51.441 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:08:51.441 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:51.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 10 librados: watch_flush enter 2024-03-25T18:08:51.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 10 librados: watch_flush exit 2024-03-25T18:08:51.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 20 client.4270.objecter shutdown clearing up homeless session... 2024-03-25T18:08:51.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 10 client.4270.objecter successfully canceled tick 2024-03-25T18:08:51.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 1 -- 172.21.15.37:0/166582887 >> v1:172.21.15.117:6800/3703993921 conn(0x7f7e6006de90 legacy=0x7f7e60070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:51.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 10 monclient: shutdown 2024-03-25T18:08:51.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:51.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 1 -- 172.21.15.37:0/166582887 >> v1:172.21.15.117:6789/0 conn(0x7f7e740a98f0 legacy=0x7f7e741376a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:51.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 1 -- 172.21.15.37:0/166582887 shutdown_connections 2024-03-25T18:08:51.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 1 -- 172.21.15.37:0/166582887 >> 172.21.15.37:0/166582887 conn(0x7f7e740b7ea0 msgr2=0x7f7e7413ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:51.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 1 -- 172.21.15.37:0/166582887 shutdown_connections 2024-03-25T18:08:51.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 1 -- 172.21.15.37:0/166582887 wait complete. 2024-03-25T18:08:51.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:51.442+0000 7f7e7a47b640 1 librados: shutdown 2024-03-25T18:08:51.454 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:58.260 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:08:58.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.376+0000 7f126a323640 10 monclient: get_monmap_and_config 2024-03-25T18:08:58.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.376+0000 7f126a323640 10 monclient: build_initial_monmap 2024-03-25T18:08:58.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.376+0000 7f126a323640 10 monclient: monmap: 2024-03-25T18:08:58.377 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:58.377 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:58.378 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:08:58.377324+0000 2024-03-25T18:08:58.378 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:08:58.377324+0000 2024-03-25T18:08:58.378 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:58.378 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:58.378 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:58.378 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:08:58.378 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:08:58.378 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:58.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.377+0000 7f126a323640 1 Processor -- start 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.377+0000 7f126a323640 1 -- start start 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.377+0000 7f126a323640 10 monclient: init 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.378+0000 7f126a323640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.378+0000 7f126a323640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.378+0000 7f126a323640 10 monclient(hunting): picked mon.noname-a con 0x7f12640a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.378+0000 7f126a323640 10 monclient(hunting): picked mon.noname-c con 0x7f126413f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.378+0000 7f126a323640 10 monclient(hunting): picked mon.noname-b con 0x7f12641430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.378+0000 7f126a323640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1264142520 con 0x7f12640a98f0 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.378+0000 7f126a323640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640b28a0 con 0x7f126413f710 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.378+0000 7f126a323640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640b2a30 con 0x7f12641430c0 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.378+0000 7f126a323640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:58.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.378+0000 7f126a323640 10 monclient(hunting): authenticate will time out at 6893.630719s 2024-03-25T18:08:58.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.386+0000 7f1263fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f12640a98f0 0x7f12640a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52910/0 (socket says 172.21.15.37:52910) 2024-03-25T18:08:58.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.386+0000 7f1263fff640 1 -- 172.21.15.37:0/1132845450 learned_addr learned my addr 172.21.15.37:0/1132845450 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:58.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.394+0000 7f1262ffd640 1 -- 172.21.15.37:0/1132845450 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3897482027 0 0) 0x7f12640b28a0 con 0x7f126413f710 2024-03-25T18:08:58.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.394+0000 7f1262ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:58.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.394+0000 7f1262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:58.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.394+0000 7f1262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:58.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.394+0000 7f1262ffd640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:08:58.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.394+0000 7f1262ffd640 1 -- 172.21.15.37:0/1132845450 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f124c003520 con 0x7f126413f710 2024-03-25T18:08:58.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.400+0000 7f1262ffd640 1 -- 172.21.15.37:0/1132845450 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637426072 0 0) 0x7f1264142520 con 0x7f12640a98f0 2024-03-25T18:08:58.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.400+0000 7f1262ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:08:58.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.400+0000 7f1262ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:58.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.400+0000 7f1262ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:58.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.400+0000 7f1262ffd640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:08:58.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.400+0000 7f1262ffd640 1 -- 172.21.15.37:0/1132845450 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12640b28a0 con 0x7f12640a98f0 2024-03-25T18:08:58.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 1 -- 172.21.15.37:0/1132845450 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1802210633 0 0) 0x7f124c003520 con 0x7f126413f710 2024-03-25T18:08:58.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 1 -- 172.21.15.37:0/1132845450 >> v1:172.21.15.37:6790/0 conn(0x7f12641430c0 legacy=0x7f126414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:58.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 1 -- 172.21.15.37:0/1132845450 >> v1:172.21.15.37:6789/0 conn(0x7f12640a98f0 legacy=0x7f12640a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:58.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 1 monclient: found mon.noname-c 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 1 -- 172.21.15.37:0/1132845450 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12640b44c0 con 0x7f126413f710 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 10 monclient: _finish_auth 0 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 1 -- 172.21.15.37:0/1132845450 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1258001690 con 0x7f126413f710 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f126a323640 5 monclient: authenticate success, global_id 4273 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f126a323640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:08:58.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f1262ffd640 10 monclient: dump: 2024-03-25T18:08:58.413 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:58.413 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:58.413 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:58.413 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:58.413 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:58.413 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:58.413 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:58.413 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:58.413 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:58.413 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:58.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.410+0000 7f126a323640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:08:58.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.419+0000 7f1262ffd640 1 -- 172.21.15.37:0/1132845450 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12580017f0 con 0x7f126413f710 2024-03-25T18:08:58.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.419+0000 7f1262ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:58.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.419+0000 7f1262ffd640 1 -- 172.21.15.37:0/1132845450 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1258001c00 con 0x7f126413f710 2024-03-25T18:08:58.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.419+0000 7f1262ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:58.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.419+0000 7f1262ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.419+0000 7f1262ffd640 10 monclient: dump: 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.419+0000 7f126a323640 10 monclient: get_monmap_and_config success 2024-03-25T18:08:58.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.419+0000 7f126a323640 4 set_mon_vals no callback set 2024-03-25T18:08:58.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.420+0000 7f126a323640 10 monclient: shutdown 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.420+0000 7f126a323640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.420+0000 7f126a323640 1 -- 172.21.15.37:0/1132845450 >> v1:172.21.15.117:6789/0 conn(0x7f126413f710 legacy=0x7f12641429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.420+0000 7f126a323640 1 -- 172.21.15.37:0/1132845450 shutdown_connections 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.420+0000 7f126a323640 1 -- 172.21.15.37:0/1132845450 >> 172.21.15.37:0/1132845450 conn(0x7f12640b7ea0 msgr2=0x7f12640b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.420+0000 7f126a323640 1 -- 172.21.15.37:0/1132845450 shutdown_connections 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.420+0000 7f126a323640 1 -- 172.21.15.37:0/1132845450 wait complete. 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.421+0000 7f126a323640 10 monclient: build_initial_monmap 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.421+0000 7f126a323640 10 monclient: monmap: 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:08:58.424 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.421+0000 7f126a323640 1 librados: starting msgr at 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.421+0000 7f126a323640 1 librados: starting objecter 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.422+0000 7f126a323640 1 Processor -- start 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.422+0000 7f126a323640 1 -- start start 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.422+0000 7f126a323640 1 librados: setting wanted keys 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.422+0000 7f126a323640 1 librados: calling monclient init 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.422+0000 7f126a323640 10 monclient: init 2024-03-25T18:08:58.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.422+0000 7f126a323640 10 monclient: _reopen_session rank -1 2024-03-25T18:08:58.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.422+0000 7f126a323640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:08:58.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.422+0000 7f126a323640 10 monclient(hunting): picked mon.noname-b con 0x7f12640a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:08:58.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.422+0000 7f126a323640 10 monclient(hunting): picked mon.noname-c con 0x7f126413f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:08:58.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.422+0000 7f126a323640 10 monclient(hunting): picked mon.noname-a con 0x7f12641430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:08:58.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.423+0000 7f126a323640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12640abdc0 con 0x7f12641430c0 2024-03-25T18:08:58.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.423+0000 7f126a323640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12641d7730 con 0x7f12640a98f0 2024-03-25T18:08:58.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.423+0000 7f126a323640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1264138f10 con 0x7f126413f710 2024-03-25T18:08:58.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.423+0000 7f126a323640 10 monclient(hunting): _renew_subs 2024-03-25T18:08:58.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.423+0000 7f126a323640 10 monclient(hunting): authenticate will time out at 6893.675367s 2024-03-25T18:08:58.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.429+0000 7f1263fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f12640a98f0 0x7f1264138800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:43406/0 (socket says 172.21.15.37:43406) 2024-03-25T18:08:58.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.429+0000 7f1263fff640 1 -- 172.21.15.37:0/3673905305 learned_addr learned my addr 172.21.15.37:0/3673905305 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:08:58.433 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.433+0000 7f1260ff9640 10 client.?.objecter ms_handle_connect 0x7f12640a98f0 2024-03-25T18:08:58.433 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.433+0000 7f1260ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:58.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.442+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3120572738 0 0) 0x7f12641d7730 con 0x7f12640a98f0 2024-03-25T18:08:58.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.442+0000 7f1260ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:58.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.442+0000 7f1260ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:58.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.442+0000 7f1260ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:58.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.442+0000 7f1260ff9640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:08:58.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.442+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1250004eb0 con 0x7f12640a98f0 2024-03-25T18:08:58.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.447+0000 7f1260ff9640 10 client.?.objecter ms_handle_connect 0x7f12641430c0 2024-03-25T18:08:58.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.447+0000 7f1260ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:58.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.451+0000 7f1260ff9640 10 client.?.objecter ms_handle_connect 0x7f126413f710 2024-03-25T18:08:58.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.451+0000 7f1260ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:08:58.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.464+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2121317319 0 0) 0x7f1250004eb0 con 0x7f12640a98f0 2024-03-25T18:08:58.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.464+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12641d7730 con 0x7f12640a98f0 2024-03-25T18:08:58.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.464+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f12540019f0 con 0x7f12640a98f0 2024-03-25T18:08:58.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.464+0000 7f1260ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:58.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.464+0000 7f1260ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:08:58.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.464+0000 7f1260ff9640 10 monclient(hunting): dump: 2024-03-25T18:08:58.465 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:58.466 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:58.466 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:58.466 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:58.466 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:58.466 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:58.466 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:58.466 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:58.466 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:58.466 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:58.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.464+0000 7f1260ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:08:58.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.467+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1995118961 0 0) 0x7f12640abdc0 con 0x7f12641430c0 2024-03-25T18:08:58.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.467+0000 7f1260ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:58.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.467+0000 7f1260ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:58.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.467+0000 7f1260ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:58.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.467+0000 7f1260ff9640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:08:58.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.467+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1250004eb0 con 0x7f12641430c0 2024-03-25T18:08:58.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.479+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1590985639 0 0) 0x7f1264138f10 con 0x7f126413f710 2024-03-25T18:08:58.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.479+0000 7f1260ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:08:58.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.479+0000 7f1260ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:08:58.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.479+0000 7f1260ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:08:58.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.479+0000 7f1260ff9640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:08:58.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.479+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12640abdc0 con 0x7f126413f710 2024-03-25T18:08:58.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3937378726 0 0) 0x7f12641d7730 con 0x7f12640a98f0 2024-03-25T18:08:58.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 >> v1:172.21.15.37:6790/0 conn(0x7f126413f710 legacy=0x7f12641d7c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:58.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 >> v1:172.21.15.37:6789/0 conn(0x7f12641430c0 legacy=0x7f12641d4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:58.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f1260ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:08:58.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f1260ff9640 1 monclient: found mon.b 2024-03-25T18:08:58.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f1260ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:08:58.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f1260ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:58.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12641390a0 con 0x7f12640a98f0 2024-03-25T18:08:58.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f1260ff9640 10 monclient: _finish_auth 0 2024-03-25T18:08:58.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f1260ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:08:58.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f1260ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:08:58.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.485+0000 7f126a323640 5 monclient: authenticate success, global_id 4276 2024-03-25T18:08:58.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.486+0000 7f126a323640 10 monclient: _renew_subs 2024-03-25T18:08:58.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.486+0000 7f126a323640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:58.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.486+0000 7f126a323640 1 -- 172.21.15.37:0/3673905305 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12641d8320 con 0x7f12640a98f0 2024-03-25T18:08:58.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.486+0000 7f126a323640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:08:58.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.486+0000 7f126a323640 10 monclient: _renew_subs 2024-03-25T18:08:58.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.486+0000 7f126a323640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:58.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.486+0000 7f126a323640 1 -- 172.21.15.37:0/3673905305 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12641d43c0 con 0x7f12640a98f0 2024-03-25T18:08:58.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.486+0000 7f126a323640 1 librados: init done 2024-03-25T18:08:58.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.487+0000 7f126a323640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:58.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.487+0000 7f126a323640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:58.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.487+0000 7f126a323640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:58.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.487+0000 7f126a323640 1 -- 172.21.15.37:0/3673905305 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12640627b0 con 0x7f12640a98f0 2024-03-25T18:08:58.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.783+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1254001c70 con 0x7f12640a98f0 2024-03-25T18:08:58.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.783+0000 7f1260ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:08:58.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.783+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1254004c30 con 0x7f12640a98f0 2024-03-25T18:08:58.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.783+0000 7f1260ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:08:58.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.783+0000 7f1260ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:08:58.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.783+0000 7f1260ff9640 10 monclient: dump: 2024-03-25T18:08:58.785 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:08:58.785 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:08:58.785 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:58.785 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:08:58.785 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:08:58.786 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:08:58.786 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:08:58.786 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:08:58.786 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:08:58.786 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:08:58.786 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.783+0000 7f12627fc640 4 set_mon_vals no callback set 2024-03-25T18:08:58.786 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.784+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f125401ac00 con 0x7f12640a98f0 2024-03-25T18:08:58.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.785+0000 7f12637fe640 10 monclient: get_auth_request con 0x7f125006de80 auth_method 0 2024-03-25T18:08:58.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.785+0000 7f1260ff9640 10 client.4276.objecter ms_handle_connect 0x7f125006de80 2024-03-25T18:08:58.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.789+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f1254054870 con 0x7f12640a98f0 2024-03-25T18:08:58.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.789+0000 7f1260ff9640 10 client.4276.objecter ms_dispatch 0x7f1264000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:08:58.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.789+0000 7f1260ff9640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:08:58.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.789+0000 7f1260ff9640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:08:58.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.789+0000 7f1260ff9640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T18:08:58.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.789+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1254050380 con 0x7f12640a98f0 2024-03-25T18:08:58.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.790+0000 7f1260ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:08:58.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:58.790+0000 7f1260ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:08:59.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.016+0000 7f126a323640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:59.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.016+0000 7f126a323640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:59.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.016+0000 7f126a323640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:08:59.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.016+0000 7f126a323640 1 -- 172.21.15.37:0/3673905305 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1264069f40 con 0x7f12640a98f0 2024-03-25T18:08:59.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.065+0000 7f1260ff9640 1 -- 172.21.15.37:0/3673905305 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f12540522b0 con 0x7f12640a98f0 2024-03-25T18:08:59.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.065+0000 7f1260ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:08:59.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.065+0000 7f1260ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:08:59.067 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:08:59.067 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:08:59.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 10 librados: watch_flush enter 2024-03-25T18:08:59.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 10 librados: watch_flush exit 2024-03-25T18:08:59.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T18:08:59.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 10 client.4276.objecter successfully canceled tick 2024-03-25T18:08:59.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 1 -- 172.21.15.37:0/3673905305 >> v1:172.21.15.117:6800/3703993921 conn(0x7f125006de80 legacy=0x7f1250070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:59.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 10 monclient: shutdown 2024-03-25T18:08:59.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:08:59.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 1 -- 172.21.15.37:0/3673905305 >> v1:172.21.15.117:6789/0 conn(0x7f12640a98f0 legacy=0x7f1264138800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:08:59.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 1 -- 172.21.15.37:0/3673905305 shutdown_connections 2024-03-25T18:08:59.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 1 -- 172.21.15.37:0/3673905305 >> 172.21.15.37:0/3673905305 conn(0x7f12640b7ea0 msgr2=0x7f1264074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:08:59.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 1 -- 172.21.15.37:0/3673905305 shutdown_connections 2024-03-25T18:08:59.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 1 -- 172.21.15.37:0/3673905305 wait complete. 2024-03-25T18:08:59.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:08:59.068+0000 7f126a323640 1 librados: shutdown 2024-03-25T18:08:59.080 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:08:59.155 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:01:34.501 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:59.158 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:01:34.504 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:59.214 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:01:34.560 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:59.221 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:01:34.567 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:59.221 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:01:34.567 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:59.230 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:01:34.543 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:59.241 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:01:34.555 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:59.274 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:01:34.619 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:08:59.285 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:01:34.598 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:05.887 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:06.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.001+0000 7f4e3087b640 10 monclient: get_monmap_and_config 2024-03-25T18:09:06.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.001+0000 7f4e3087b640 10 monclient: build_initial_monmap 2024-03-25T18:09:06.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.001+0000 7f4e3087b640 10 monclient: monmap: 2024-03-25T18:09:06.002 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:06.002 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:06.003 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:09:06.002752+0000 2024-03-25T18:09:06.003 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:09:06.002752+0000 2024-03-25T18:09:06.003 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:06.003 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:06.003 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:06.003 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:09:06.003 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:09:06.003 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.002+0000 7f4e3087b640 1 Processor -- start 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.002+0000 7f4e3087b640 1 -- start start 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.002+0000 7f4e3087b640 10 monclient: init 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.003+0000 7f4e3087b640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.003+0000 7f4e3087b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.003+0000 7f4e3087b640 10 monclient(hunting): picked mon.noname-c con 0x7f4e2c0a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.003+0000 7f4e3087b640 10 monclient(hunting): picked mon.noname-a con 0x7f4e2c13f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.003+0000 7f4e3087b640 10 monclient(hunting): picked mon.noname-b con 0x7f4e2c1430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.003+0000 7f4e3087b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c142520 con 0x7f4e2c13f710 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.003+0000 7f4e3087b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c0b28a0 con 0x7f4e2c0a98f0 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.003+0000 7f4e3087b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c0b2a30 con 0x7f4e2c1430c0 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.003+0000 7f4e3087b640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.003+0000 7f4e3087b640 10 monclient(hunting): authenticate will time out at 6901.255923s 2024-03-25T18:09:06.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.008+0000 7f4e29d9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f4e2c13f710 0x7f4e2c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:57454/0 (socket says 172.21.15.37:57454) 2024-03-25T18:09:06.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.008+0000 7f4e29d9b640 1 -- 172.21.15.37:0/2087892536 learned_addr learned my addr 172.21.15.37:0/2087892536 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:06.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.020+0000 7f4e2959a640 1 -- 172.21.15.37:0/2087892536 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 342728719 0 0) 0x7f4e2c142520 con 0x7f4e2c13f710 2024-03-25T18:09:06.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.020+0000 7f4e2959a640 10 monclient(hunting): _negotiate 2024-03-25T18:09:06.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.020+0000 7f4e2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:06.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.020+0000 7f4e2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:06.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.020+0000 7f4e2959a640 10 monclient(hunting): my global_id is 4275 2024-03-25T18:09:06.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.020+0000 7f4e2959a640 1 -- 172.21.15.37:0/2087892536 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e14003520 con 0x7f4e2c13f710 2024-03-25T18:09:06.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.027+0000 7f4e2959a640 1 -- 172.21.15.37:0/2087892536 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496554449 0 0) 0x7f4e2c0b28a0 con 0x7f4e2c0a98f0 2024-03-25T18:09:06.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.027+0000 7f4e2959a640 10 monclient(hunting): _negotiate 2024-03-25T18:09:06.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.027+0000 7f4e2959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:06.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.027+0000 7f4e2959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:06.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.027+0000 7f4e2959a640 10 monclient(hunting): my global_id is 4279 2024-03-25T18:09:06.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.027+0000 7f4e2959a640 1 -- 172.21.15.37:0/2087892536 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e2c142520 con 0x7f4e2c0a98f0 2024-03-25T18:09:06.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 1 -- 172.21.15.37:0/2087892536 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 118036372 0 0) 0x7f4e14003520 con 0x7f4e2c13f710 2024-03-25T18:09:06.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 1 -- 172.21.15.37:0/2087892536 >> v1:172.21.15.37:6790/0 conn(0x7f4e2c1430c0 legacy=0x7f4e2c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:06.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 1 -- 172.21.15.37:0/2087892536 >> v1:172.21.15.117:6789/0 conn(0x7f4e2c0a98f0 legacy=0x7f4e2c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:06.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 10 monclient: _finish_hunting 0 2024-03-25T18:09:06.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 1 monclient: found mon.noname-a 2024-03-25T18:09:06.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:06.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 1 -- 172.21.15.37:0/2087892536 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c0b44c0 con 0x7f4e2c13f710 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 10 monclient: _finish_auth 0 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 10 monclient: _check_auth_tickets 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e3087b640 5 monclient: authenticate success, global_id 4275 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 1 -- 172.21.15.37:0/2087892536 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f4e20001690 con 0x7f4e2c13f710 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e2959a640 10 monclient: dump: 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:06.038 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:06.039 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:06.039 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:06.039 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:06.039 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:06.039 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:06.039 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:06.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.036+0000 7f4e3087b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:06.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.046+0000 7f4e2959a640 1 -- 172.21.15.37:0/2087892536 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e200017f0 con 0x7f4e2c13f710 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.046+0000 7f4e2959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.046+0000 7f4e2959a640 1 -- 172.21.15.37:0/2087892536 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f4e20001c00 con 0x7f4e2c13f710 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.046+0000 7f4e3087b640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.046+0000 7f4e3087b640 4 set_mon_vals no callback set 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.046+0000 7f4e2959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.046+0000 7f4e2959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.047+0000 7f4e2959a640 10 monclient: dump: 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:06.048 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:06.049 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:06.049 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:06.049 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.047+0000 7f4e3087b640 10 monclient: shutdown 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.047+0000 7f4e3087b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.047+0000 7f4e3087b640 1 -- 172.21.15.37:0/2087892536 >> v1:172.21.15.37:6789/0 conn(0x7f4e2c13f710 legacy=0x7f4e2c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.047+0000 7f4e3087b640 1 -- 172.21.15.37:0/2087892536 shutdown_connections 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.047+0000 7f4e3087b640 1 -- 172.21.15.37:0/2087892536 >> 172.21.15.37:0/2087892536 conn(0x7f4e2c0b7ea0 msgr2=0x7f4e2c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.047+0000 7f4e3087b640 1 -- 172.21.15.37:0/2087892536 shutdown_connections 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.047+0000 7f4e3087b640 1 -- 172.21.15.37:0/2087892536 wait complete. 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.048+0000 7f4e3087b640 10 monclient: build_initial_monmap 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.048+0000 7f4e3087b640 10 monclient: monmap: 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:06.051 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.049+0000 7f4e3087b640 1 librados: starting msgr at 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.049+0000 7f4e3087b640 1 librados: starting objecter 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.049+0000 7f4e3087b640 1 Processor -- start 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.049+0000 7f4e3087b640 1 -- start start 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.049+0000 7f4e3087b640 1 librados: setting wanted keys 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.049+0000 7f4e3087b640 1 librados: calling monclient init 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.049+0000 7f4e3087b640 10 monclient: init 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.050+0000 7f4e3087b640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.050+0000 7f4e3087b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.050+0000 7f4e3087b640 10 monclient(hunting): picked mon.noname-a con 0x7f4e2c0a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.050+0000 7f4e3087b640 10 monclient(hunting): picked mon.noname-b con 0x7f4e2c13f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.050+0000 7f4e3087b640 10 monclient(hunting): picked mon.noname-c con 0x7f4e2c1430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.050+0000 7f4e3087b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c139520 con 0x7f4e2c0a98f0 2024-03-25T18:09:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.050+0000 7f4e3087b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c138b40 con 0x7f4e2c13f710 2024-03-25T18:09:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.050+0000 7f4e3087b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e2c138e20 con 0x7f4e2c1430c0 2024-03-25T18:09:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.050+0000 7f4e3087b640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.050+0000 7f4e3087b640 10 monclient(hunting): authenticate will time out at 6901.302708s 2024-03-25T18:09:06.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.061+0000 7f4e2ad9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f4e2c1430c0 0x7f4e2c1e2290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:55528/0 (socket says 172.21.15.37:55528) 2024-03-25T18:09:06.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.061+0000 7f4e2ad9d640 1 -- 172.21.15.37:0/3853467217 learned_addr learned my addr 172.21.15.37:0/3853467217 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:06.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.063+0000 7f4e1affd640 10 client.?.objecter ms_handle_connect 0x7f4e2c1430c0 2024-03-25T18:09:06.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.063+0000 7f4e1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:06.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.065+0000 7f4e1affd640 10 client.?.objecter ms_handle_connect 0x7f4e2c13f710 2024-03-25T18:09:06.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.065+0000 7f4e1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:06.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.075+0000 7f4e1affd640 10 client.?.objecter ms_handle_connect 0x7f4e2c0a98f0 2024-03-25T18:09:06.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.075+0000 7f4e1affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:06.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.076+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403186139 0 0) 0x7f4e2c138b40 con 0x7f4e2c13f710 2024-03-25T18:09:06.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.076+0000 7f4e1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:06.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.076+0000 7f4e1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:06.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.076+0000 7f4e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:06.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.076+0000 7f4e1affd640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:09:06.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.076+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e0c004eb0 con 0x7f4e2c13f710 2024-03-25T18:09:06.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.079+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491446044 0 0) 0x7f4e2c138e20 con 0x7f4e2c1430c0 2024-03-25T18:09:06.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.079+0000 7f4e1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:06.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.079+0000 7f4e1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:06.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.079+0000 7f4e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:06.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.079+0000 7f4e1affd640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:09:06.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.079+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e2c138b40 con 0x7f4e2c1430c0 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300405993 0 0) 0x7f4e2c139520 con 0x7f4e2c0a98f0 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4e2c138e20 con 0x7f4e2c0a98f0 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1909619754 0 0) 0x7f4e0c004eb0 con 0x7f4e2c13f710 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e2c139520 con 0x7f4e2c13f710 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f4e20001b00 con 0x7f4e2c13f710 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 10 monclient(hunting): dump: 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:06.095 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:06.096 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:06.096 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:06.096 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:06.096 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:06.096 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:06.096 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:06.096 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:06.096 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:06.096 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.093+0000 7f4e1affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:06.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.096+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2268499343 0 0) 0x7f4e2c138b40 con 0x7f4e2c1430c0 2024-03-25T18:09:06.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.096+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4e0c004eb0 con 0x7f4e2c1430c0 2024-03-25T18:09:06.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.096+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f4e24001c10 con 0x7f4e2c1430c0 2024-03-25T18:09:06.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.096+0000 7f4e1affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:06.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.096+0000 7f4e1affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:06.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.096+0000 7f4e1affd640 10 monclient(hunting): dump: 2024-03-25T18:09:06.097 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:06.097 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:06.097 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:06.097 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:06.098 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:06.098 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:06.098 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:06.098 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:06.098 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:06.098 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1117659206 0 0) 0x7f4e0c004eb0 con 0x7f4e2c1430c0 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 >> v1:172.21.15.117:6789/0 conn(0x7f4e2c13f710 legacy=0x7f4e2c137b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 >> v1:172.21.15.37:6789/0 conn(0x7f4e2c0a98f0 legacy=0x7f4e2c137460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e1affd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e1affd640 1 monclient: found mon.c 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e1affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e1affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e2c139100 con 0x7f4e2c1430c0 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e1affd640 10 monclient: _finish_auth 0 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e1affd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e1affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:06.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e3087b640 5 monclient: authenticate success, global_id 4277 2024-03-25T18:09:06.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e3087b640 10 monclient: _renew_subs 2024-03-25T18:09:06.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e3087b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:06.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e3087b640 1 -- 172.21.15.37:0/3853467217 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e2c136aa0 con 0x7f4e2c1430c0 2024-03-25T18:09:06.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e3087b640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:06.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e3087b640 10 monclient: _renew_subs 2024-03-25T18:09:06.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e3087b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:06.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.106+0000 7f4e3087b640 1 -- 172.21.15.37:0/3853467217 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e2c13ba10 con 0x7f4e2c1430c0 2024-03-25T18:09:06.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.107+0000 7f4e3087b640 1 librados: init done 2024-03-25T18:09:06.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.107+0000 7f4e3087b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:06.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.107+0000 7f4e3087b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:06.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.107+0000 7f4e3087b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:06.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.107+0000 7f4e3087b640 1 -- 172.21.15.37:0/3853467217 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e2c136c90 con 0x7f4e2c1430c0 2024-03-25T18:09:06.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.412+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4e24003970 con 0x7f4e2c1430c0 2024-03-25T18:09:06.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.412+0000 7f4e1affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:06.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.412+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f4e24002100 con 0x7f4e2c1430c0 2024-03-25T18:09:06.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.412+0000 7f4e1affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:06.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.412+0000 7f4e1affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:06.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.412+0000 7f4e1affd640 10 monclient: dump: 2024-03-25T18:09:06.413 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:06.413 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:06.413 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:06.413 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:06.414 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:06.414 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:06.414 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:06.414 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:06.414 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:06.414 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:06.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.412+0000 7f4e28d99640 4 set_mon_vals no callback set 2024-03-25T18:09:06.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.413+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f4e24003d80 con 0x7f4e2c1430c0 2024-03-25T18:09:06.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.413+0000 7f4e2a59c640 10 monclient: get_auth_request con 0x7f4e0c06dfb0 auth_method 0 2024-03-25T18:09:06.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.414+0000 7f4e1affd640 10 client.4277.objecter ms_handle_connect 0x7f4e0c06dfb0 2024-03-25T18:09:06.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.420+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f4e24054a60 con 0x7f4e2c1430c0 2024-03-25T18:09:06.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.420+0000 7f4e1affd640 10 client.4277.objecter ms_dispatch 0x7f4e2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:06.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.420+0000 7f4e1affd640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:06.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.420+0000 7f4e1affd640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:06.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.420+0000 7f4e1affd640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-25T18:09:06.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.420+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4e24050570 con 0x7f4e2c1430c0 2024-03-25T18:09:06.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.420+0000 7f4e1affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:06.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.420+0000 7f4e1affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:06.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.660+0000 7f4e3087b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:06.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.660+0000 7f4e3087b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:06.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.660+0000 7f4e3087b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:06.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.660+0000 7f4e3087b640 1 -- 172.21.15.37:0/3853467217 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4e2c0627b0 con 0x7f4e2c1430c0 2024-03-25T18:09:06.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.691+0000 7f4e1affd640 1 -- 172.21.15.37:0/3853467217 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f4e240524a0 con 0x7f4e2c1430c0 2024-03-25T18:09:06.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.691+0000 7f4e1affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:06.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.691+0000 7f4e1affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:06.692 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:09:06.692 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:06.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 10 librados: watch_flush enter 2024-03-25T18:09:06.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 10 librados: watch_flush exit 2024-03-25T18:09:06.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-25T18:09:06.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 10 client.4277.objecter successfully canceled tick 2024-03-25T18:09:06.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 1 -- 172.21.15.37:0/3853467217 >> v1:172.21.15.117:6800/3703993921 conn(0x7f4e0c06dfb0 legacy=0x7f4e0c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:06.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 10 monclient: shutdown 2024-03-25T18:09:06.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:06.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 1 -- 172.21.15.37:0/3853467217 >> v1:172.21.15.37:6790/0 conn(0x7f4e2c1430c0 legacy=0x7f4e2c1e2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:06.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 1 -- 172.21.15.37:0/3853467217 shutdown_connections 2024-03-25T18:09:06.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 1 -- 172.21.15.37:0/3853467217 >> 172.21.15.37:0/3853467217 conn(0x7f4e2c0b7ea0 msgr2=0x7f4e2c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:06.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 1 -- 172.21.15.37:0/3853467217 shutdown_connections 2024-03-25T18:09:06.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 1 -- 172.21.15.37:0/3853467217 wait complete. 2024-03-25T18:09:06.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:06.693+0000 7f4e3087b640 1 librados: shutdown 2024-03-25T18:09:06.707 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:13.513 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:13.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.628+0000 7f837eefb640 10 monclient: get_monmap_and_config 2024-03-25T18:09:13.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.628+0000 7f837eefb640 10 monclient: build_initial_monmap 2024-03-25T18:09:13.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.628+0000 7f837eefb640 10 monclient: monmap: 2024-03-25T18:09:13.629 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:13.630 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:13.630 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:09:13.629548+0000 2024-03-25T18:09:13.630 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:09:13.629548+0000 2024-03-25T18:09:13.630 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:13.630 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:13.630 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:13.630 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:09:13.630 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:09:13.630 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.629+0000 7f837eefb640 1 Processor -- start 2024-03-25T18:09:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.629+0000 7f837eefb640 1 -- start start 2024-03-25T18:09:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.629+0000 7f837eefb640 10 monclient: init 2024-03-25T18:09:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.629+0000 7f837eefb640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.629+0000 7f837eefb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.629+0000 7f837eefb640 10 monclient(hunting): picked mon.noname-c con 0x7f83780a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.630+0000 7f837eefb640 10 monclient(hunting): picked mon.noname-b con 0x7f837813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:13.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.630+0000 7f837eefb640 10 monclient(hunting): picked mon.noname-a con 0x7f83781430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:13.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.630+0000 7f837eefb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8378142520 con 0x7f83781430c0 2024-03-25T18:09:13.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.630+0000 7f837eefb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83780b28a0 con 0x7f83780a98f0 2024-03-25T18:09:13.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.630+0000 7f837eefb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83780b2a30 con 0x7f837813f710 2024-03-25T18:09:13.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.630+0000 7f837eefb640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:13.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.630+0000 7f837eefb640 10 monclient(hunting): authenticate will time out at 6908.882345s 2024-03-25T18:09:13.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.647+0000 7f837cc97640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f83780a98f0 0x7f83780a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:54424/0 (socket says 172.21.15.37:54424) 2024-03-25T18:09:13.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.647+0000 7f837cc97640 1 -- 172.21.15.37:0/3584850222 learned_addr learned my addr 172.21.15.37:0/3584850222 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:13.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.657+0000 7f83777fe640 1 -- 172.21.15.37:0/3584850222 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285196789 0 0) 0x7f83780b28a0 con 0x7f83780a98f0 2024-03-25T18:09:13.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.657+0000 7f83777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:13.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.657+0000 7f83777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:13.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.657+0000 7f83777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:13.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.657+0000 7f83777fe640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:09:13.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.657+0000 7f83777fe640 1 -- 172.21.15.37:0/3584850222 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8368003520 con 0x7f83780a98f0 2024-03-25T18:09:13.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 1 -- 172.21.15.37:0/3584850222 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4072060407 0 0) 0x7f8368003520 con 0x7f83780a98f0 2024-03-25T18:09:13.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 1 -- 172.21.15.37:0/3584850222 >> v1:172.21.15.37:6790/0 conn(0x7f837813f710 legacy=0x7f83781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:13.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 1 -- 172.21.15.37:0/3584850222 >> v1:172.21.15.37:6789/0 conn(0x7f83781430c0 legacy=0x7f837814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:13.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:13.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 1 monclient: found mon.noname-c 2024-03-25T18:09:13.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:13.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:09:13.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 1 -- 172.21.15.37:0/3584850222 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83780b44c0 con 0x7f83780a98f0 2024-03-25T18:09:13.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:13.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:13.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 1 -- 172.21.15.37:0/3584850222 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f836c001690 con 0x7f83780a98f0 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f83777fe640 10 monclient: dump: 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:13.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f837eefb640 5 monclient: authenticate success, global_id 4285 2024-03-25T18:09:13.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.669+0000 7f837eefb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:13.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.696+0000 7f83777fe640 1 -- 172.21.15.37:0/3584850222 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f836c0017f0 con 0x7f83780a98f0 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.696+0000 7f83777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.696+0000 7f83777fe640 1 -- 172.21.15.37:0/3584850222 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f836c001c00 con 0x7f83780a98f0 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.696+0000 7f83777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.696+0000 7f83777fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.696+0000 7f83777fe640 10 monclient: dump: 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:13.697 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:13.698 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:13.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.696+0000 7f837eefb640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:13.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.696+0000 7f837eefb640 4 set_mon_vals no callback set 2024-03-25T18:09:13.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.697+0000 7f837eefb640 10 monclient: shutdown 2024-03-25T18:09:13.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.697+0000 7f837eefb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:13.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.697+0000 7f837eefb640 1 -- 172.21.15.37:0/3584850222 >> v1:172.21.15.117:6789/0 conn(0x7f83780a98f0 legacy=0x7f83780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:13.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.697+0000 7f837eefb640 1 -- 172.21.15.37:0/3584850222 shutdown_connections 2024-03-25T18:09:13.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.697+0000 7f837eefb640 1 -- 172.21.15.37:0/3584850222 >> 172.21.15.37:0/3584850222 conn(0x7f83780b7ea0 msgr2=0x7f83780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:13.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.697+0000 7f837eefb640 1 -- 172.21.15.37:0/3584850222 shutdown_connections 2024-03-25T18:09:13.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.697+0000 7f837eefb640 1 -- 172.21.15.37:0/3584850222 wait complete. 2024-03-25T18:09:13.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.698+0000 7f837eefb640 10 monclient: build_initial_monmap 2024-03-25T18:09:13.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.698+0000 7f837eefb640 10 monclient: monmap: 2024-03-25T18:09:13.699 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.698+0000 7f837eefb640 1 librados: starting msgr at 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.698+0000 7f837eefb640 1 librados: starting objecter 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.698+0000 7f837eefb640 1 Processor -- start 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.698+0000 7f837eefb640 1 -- start start 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.698+0000 7f837eefb640 1 librados: setting wanted keys 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.698+0000 7f837eefb640 1 librados: calling monclient init 2024-03-25T18:09:13.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.698+0000 7f837eefb640 10 monclient: init 2024-03-25T18:09:13.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.699+0000 7f837eefb640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:13.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.699+0000 7f837eefb640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:09:13.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.699+0000 7f837eefb640 10 monclient(hunting): picked mon.noname-a con 0x7f83780a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:13.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.699+0000 7f837eefb640 10 monclient(hunting): picked mon.noname-b con 0x7f837813f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:13.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.699+0000 7f837eefb640 10 monclient(hunting): picked mon.noname-c con 0x7f83781430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:13.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.699+0000 7f837eefb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83781374f0 con 0x7f83780a98f0 2024-03-25T18:09:13.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.699+0000 7f837eefb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83781db360 con 0x7f837813f710 2024-03-25T18:09:13.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.699+0000 7f837eefb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8378133a40 con 0x7f83781430c0 2024-03-25T18:09:13.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.699+0000 7f837eefb640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:13.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.699+0000 7f837eefb640 10 monclient(hunting): authenticate will time out at 6908.951625s 2024-03-25T18:09:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.704+0000 7f837d498640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f83781430c0 0x7f8378133330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:46608/0 (socket says 172.21.15.37:46608) 2024-03-25T18:09:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.704+0000 7f837d498640 1 -- 172.21.15.37:0/2112413454 learned_addr learned my addr 172.21.15.37:0/2112413454 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:13.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.706+0000 7f83757fa640 10 client.?.objecter ms_handle_connect 0x7f83781430c0 2024-03-25T18:09:13.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.706+0000 7f83757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:13.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.708+0000 7f83757fa640 10 client.?.objecter ms_handle_connect 0x7f837813f710 2024-03-25T18:09:13.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.708+0000 7f83757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:13.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.713+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3291238498 0 0) 0x7f8378133a40 con 0x7f83781430c0 2024-03-25T18:09:13.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.713+0000 7f83757fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:13.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.713+0000 7f83757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:13.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.713+0000 7f83757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:13.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.713+0000 7f83757fa640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:09:13.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.713+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f835c004eb0 con 0x7f83781430c0 2024-03-25T18:09:13.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.714+0000 7f83757fa640 10 client.?.objecter ms_handle_connect 0x7f83780a98f0 2024-03-25T18:09:13.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.714+0000 7f83757fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:13.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.717+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 378119696 0 0) 0x7f83781db360 con 0x7f837813f710 2024-03-25T18:09:13.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.717+0000 7f83757fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:13.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.717+0000 7f83757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:13.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.717+0000 7f83757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:13.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.717+0000 7f83757fa640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:09:13.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.717+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8378133a40 con 0x7f837813f710 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.722+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3950419407 0 0) 0x7f83781374f0 con 0x7f83780a98f0 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83781db360 con 0x7f83780a98f0 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2811563018 0 0) 0x7f835c004eb0 con 0x7f83781430c0 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83781374f0 con 0x7f83781430c0 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8370001c90 con 0x7f83781430c0 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:13.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 10 monclient(hunting): dump: 2024-03-25T18:09:13.725 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:13.725 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:13.725 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:13.725 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:13.725 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:13.725 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:13.725 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:13.725 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:13.725 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:13.725 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:13.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.723+0000 7f83757fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:13.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.733+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4145271416 0 0) 0x7f8378133a40 con 0x7f837813f710 2024-03-25T18:09:13.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.733+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f835c004eb0 con 0x7f837813f710 2024-03-25T18:09:13.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.733+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f83640018a0 con 0x7f837813f710 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.733+0000 7f83757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.733+0000 7f83757fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.733+0000 7f83757fa640 10 monclient(hunting): dump: 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:13.735 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:13.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.734+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2156800125 0 0) 0x7f83781374f0 con 0x7f83781430c0 2024-03-25T18:09:13.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.734+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 >> v1:172.21.15.117:6789/0 conn(0x7f837813f710 legacy=0x7f8378132c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:13.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.734+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 >> v1:172.21.15.37:6789/0 conn(0x7f83780a98f0 legacy=0x7f83781dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:13.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f83757fa640 10 monclient: _finish_hunting 0 2024-03-25T18:09:13.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f83757fa640 1 monclient: found mon.c 2024-03-25T18:09:13.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f83757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:13.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f83757fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:13.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8378138000 con 0x7f83781430c0 2024-03-25T18:09:13.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f83757fa640 10 monclient: _finish_auth 0 2024-03-25T18:09:13.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f83757fa640 10 monclient: _check_auth_tickets 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f83757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f837eefb640 5 monclient: authenticate success, global_id 4283 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f837eefb640 10 monclient: _renew_subs 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f837eefb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f837eefb640 1 -- 172.21.15.37:0/2112413454 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8378136030 con 0x7f83781430c0 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f837eefb640 10 client.4283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f837eefb640 10 monclient: _renew_subs 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f837eefb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f837eefb640 1 -- 172.21.15.37:0/2112413454 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83780b7570 con 0x7f83781430c0 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.735+0000 7f837eefb640 1 librados: init done 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.736+0000 7f837eefb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.736+0000 7f837eefb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.736+0000 7f837eefb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:13.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:13.736+0000 7f837eefb640 1 -- 172.21.15.37:0/2112413454 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83780627b0 con 0x7f83781430c0 2024-03-25T18:09:14.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.046+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8370003990 con 0x7f83781430c0 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.046+0000 7f83757fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.046+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8370003d60 con 0x7f83781430c0 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.046+0000 7f83757fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.046+0000 7f83757fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.046+0000 7f83757fa640 10 monclient: dump: 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:14.048 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:14.049 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:14.049 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:14.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.046+0000 7f8376ffd640 4 set_mon_vals no callback set 2024-03-25T18:09:14.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.047+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f8370003f80 con 0x7f83781430c0 2024-03-25T18:09:14.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.048+0000 7f837cc97640 10 monclient: get_auth_request con 0x7f835c06df70 auth_method 0 2024-03-25T18:09:14.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.048+0000 7f83757fa640 10 client.4283.objecter ms_handle_connect 0x7f835c06df70 2024-03-25T18:09:14.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.054+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f8370054b00 con 0x7f83781430c0 2024-03-25T18:09:14.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.054+0000 7f83757fa640 10 client.4283.objecter ms_dispatch 0x7f8378000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:14.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.054+0000 7f83757fa640 3 client.4283.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:14.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.054+0000 7f83757fa640 3 client.4283.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:14.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.054+0000 7f83757fa640 20 client.4283.objecter dump_active .. 0 homeless 2024-03-25T18:09:14.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.054+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8370050610 con 0x7f83781430c0 2024-03-25T18:09:14.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.054+0000 7f83757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:14.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.054+0000 7f83757fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:14.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.276+0000 7f837eefb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:14.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.276+0000 7f837eefb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:14.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.276+0000 7f837eefb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:14.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.276+0000 7f837eefb640 1 -- 172.21.15.37:0/2112413454 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8378136270 con 0x7f83781430c0 2024-03-25T18:09:14.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.298+0000 7f83757fa640 1 -- 172.21.15.37:0/2112413454 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f8370052540 con 0x7f83781430c0 2024-03-25T18:09:14.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.298+0000 7f83757fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:14.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.298+0000 7f83757fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:14.299 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:09:14.299 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:14.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 10 librados: watch_flush enter 2024-03-25T18:09:14.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 10 librados: watch_flush exit 2024-03-25T18:09:14.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 20 client.4283.objecter shutdown clearing up homeless session... 2024-03-25T18:09:14.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 10 client.4283.objecter successfully canceled tick 2024-03-25T18:09:14.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 1 -- 172.21.15.37:0/2112413454 >> v1:172.21.15.117:6800/3703993921 conn(0x7f835c06df70 legacy=0x7f835c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:14.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 10 monclient: shutdown 2024-03-25T18:09:14.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:14.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 1 -- 172.21.15.37:0/2112413454 >> v1:172.21.15.37:6790/0 conn(0x7f83781430c0 legacy=0x7f8378133330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:14.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 1 -- 172.21.15.37:0/2112413454 shutdown_connections 2024-03-25T18:09:14.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 1 -- 172.21.15.37:0/2112413454 >> 172.21.15.37:0/2112413454 conn(0x7f83780b7ea0 msgr2=0x7f837813d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:14.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 1 -- 172.21.15.37:0/2112413454 shutdown_connections 2024-03-25T18:09:14.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 1 -- 172.21.15.37:0/2112413454 wait complete. 2024-03-25T18:09:14.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:14.301+0000 7f837eefb640 1 librados: shutdown 2024-03-25T18:09:14.314 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:19.193 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:01:54.539 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:19.249 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:01:54.595 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:19.295 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:01:54.640 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:19.297 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:01:54.643 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:19.306 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:01:54.652 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:19.320 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:01:54.633 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:19.327 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:01:54.641 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:19.341 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:01:54.686 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:19.348 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:01:54.661 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:21.120 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:21.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.234+0000 7fb0f0025640 10 monclient: get_monmap_and_config 2024-03-25T18:09:21.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.234+0000 7fb0f0025640 10 monclient: build_initial_monmap 2024-03-25T18:09:21.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.234+0000 7fb0f0025640 10 monclient: monmap: 2024-03-25T18:09:21.235 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:21.235 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:21.235 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:09:21.235560+0000 2024-03-25T18:09:21.235 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:09:21.235560+0000 2024-03-25T18:09:21.235 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:21.235 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:21.236 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:21.236 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:09:21.236 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:09:21.236 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:21.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.235+0000 7fb0f0025640 1 Processor -- start 2024-03-25T18:09:21.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.235+0000 7fb0f0025640 1 -- start start 2024-03-25T18:09:21.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.235+0000 7fb0f0025640 10 monclient: init 2024-03-25T18:09:21.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.236+0000 7fb0f0025640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:21.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.236+0000 7fb0f0025640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:21.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.236+0000 7fb0f0025640 10 monclient(hunting): picked mon.noname-a con 0x7fb0e80a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:21.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.236+0000 7fb0f0025640 10 monclient(hunting): picked mon.noname-b con 0x7fb0e813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:21.237 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.236+0000 7fb0f0025640 10 monclient(hunting): picked mon.noname-c con 0x7fb0e81430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:21.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.236+0000 7fb0f0025640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e8142520 con 0x7fb0e80a98f0 2024-03-25T18:09:21.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.236+0000 7fb0f0025640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e80b28a0 con 0x7fb0e81430c0 2024-03-25T18:09:21.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.236+0000 7fb0f0025640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e80b2a30 con 0x7fb0e813f710 2024-03-25T18:09:21.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.236+0000 7fb0f0025640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:21.238 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.236+0000 7fb0f0025640 10 monclient(hunting): authenticate will time out at 6916.488880s 2024-03-25T18:09:21.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.250+0000 7fb0ed5c0640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fb0e813f710 0x7fb0e81429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:49062/0 (socket says 172.21.15.37:49062) 2024-03-25T18:09:21.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.250+0000 7fb0ed5c0640 1 -- 172.21.15.37:0/2755325922 learned_addr learned my addr 172.21.15.37:0/2755325922 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.261+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1866486011 0 0) 0x7fb0e80b28a0 con 0x7fb0e81430c0 2024-03-25T18:09:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.261+0000 7fb0ecdbf640 10 monclient(hunting): _negotiate 2024-03-25T18:09:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.261+0000 7fb0ecdbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.261+0000 7fb0ecdbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.261+0000 7fb0ecdbf640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:09:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.261+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0d4003520 con 0x7fb0e81430c0 2024-03-25T18:09:21.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.265+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 936917333 0 0) 0x7fb0e80b2a30 con 0x7fb0e813f710 2024-03-25T18:09:21.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.265+0000 7fb0ecdbf640 10 monclient(hunting): _negotiate 2024-03-25T18:09:21.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.265+0000 7fb0ecdbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:21.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.265+0000 7fb0ecdbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:21.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.265+0000 7fb0ecdbf640 10 monclient(hunting): my global_id is 4286 2024-03-25T18:09:21.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.265+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e80b28a0 con 0x7fb0e813f710 2024-03-25T18:09:21.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.275+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 345449025 0 0) 0x7fb0e8142520 con 0x7fb0e80a98f0 2024-03-25T18:09:21.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.275+0000 7fb0ecdbf640 10 monclient(hunting): _negotiate 2024-03-25T18:09:21.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.275+0000 7fb0ecdbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:21.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.275+0000 7fb0ecdbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:21.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.275+0000 7fb0ecdbf640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:09:21.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.275+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e80b2a30 con 0x7fb0e80a98f0 2024-03-25T18:09:21.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.278+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4168131009 0 0) 0x7fb0e80b28a0 con 0x7fb0e813f710 2024-03-25T18:09:21.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.278+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 >> v1:172.21.15.117:6789/0 conn(0x7fb0e81430c0 legacy=0x7fb0e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:21.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.278+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 >> v1:172.21.15.37:6789/0 conn(0x7fb0e80a98f0 legacy=0x7fb0e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:21.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.278+0000 7fb0ecdbf640 10 monclient: _finish_hunting 0 2024-03-25T18:09:21.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.278+0000 7fb0ecdbf640 1 monclient: found mon.noname-b 2024-03-25T18:09:21.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.278+0000 7fb0ecdbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:21.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.278+0000 7fb0ecdbf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.279+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e80b44c0 con 0x7fb0e813f710 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.279+0000 7fb0ecdbf640 10 monclient: _finish_auth 0 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.279+0000 7fb0ecdbf640 10 monclient: _check_auth_tickets 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.279+0000 7fb0ecdbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.279+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb0d8001670 con 0x7fb0e813f710 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.279+0000 7fb0ecdbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.279+0000 7fb0ecdbf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.279+0000 7fb0ecdbf640 10 monclient: dump: 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:21.281 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:21.282 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:21.282 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:21.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.279+0000 7fb0f0025640 5 monclient: authenticate success, global_id 4286 2024-03-25T18:09:21.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.279+0000 7fb0f0025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:21.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.291+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0d80017d0 con 0x7fb0e813f710 2024-03-25T18:09:21.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.291+0000 7fb0ecdbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:21.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.291+0000 7fb0ecdbf640 1 -- 172.21.15.37:0/2755325922 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb0d8001be0 con 0x7fb0e813f710 2024-03-25T18:09:21.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.291+0000 7fb0ecdbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:21.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.291+0000 7fb0ecdbf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:21.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.291+0000 7fb0ecdbf640 10 monclient: dump: 2024-03-25T18:09:21.292 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:21.292 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:21.292 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:21.292 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:21.293 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:21.293 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:21.293 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:21.293 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:21.293 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:21.293 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:21.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.291+0000 7fb0f0025640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:21.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.291+0000 7fb0f0025640 4 set_mon_vals no callback set 2024-03-25T18:09:21.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.292+0000 7fb0f0025640 10 monclient: shutdown 2024-03-25T18:09:21.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.292+0000 7fb0f0025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:21.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.292+0000 7fb0f0025640 1 -- 172.21.15.37:0/2755325922 >> v1:172.21.15.37:6790/0 conn(0x7fb0e813f710 legacy=0x7fb0e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:21.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.292+0000 7fb0f0025640 1 -- 172.21.15.37:0/2755325922 shutdown_connections 2024-03-25T18:09:21.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.292+0000 7fb0f0025640 1 -- 172.21.15.37:0/2755325922 >> 172.21.15.37:0/2755325922 conn(0x7fb0e80b7ea0 msgr2=0x7fb0e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:21.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.292+0000 7fb0f0025640 1 -- 172.21.15.37:0/2755325922 shutdown_connections 2024-03-25T18:09:21.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.292+0000 7fb0f0025640 1 -- 172.21.15.37:0/2755325922 wait complete. 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.293+0000 7fb0f0025640 10 monclient: build_initial_monmap 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.293+0000 7fb0f0025640 10 monclient: monmap: 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.293+0000 7fb0f0025640 1 librados: starting msgr at 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.293+0000 7fb0f0025640 1 librados: starting objecter 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.293+0000 7fb0f0025640 1 Processor -- start 2024-03-25T18:09:21.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.293+0000 7fb0f0025640 1 -- start start 2024-03-25T18:09:21.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.293+0000 7fb0f0025640 1 librados: setting wanted keys 2024-03-25T18:09:21.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.293+0000 7fb0f0025640 1 librados: calling monclient init 2024-03-25T18:09:21.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.293+0000 7fb0f0025640 10 monclient: init 2024-03-25T18:09:21.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.294+0000 7fb0f0025640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:21.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.294+0000 7fb0f0025640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:21.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.294+0000 7fb0f0025640 10 monclient(hunting): picked mon.noname-c con 0x7fb0e80a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:21.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.294+0000 7fb0f0025640 10 monclient(hunting): picked mon.noname-a con 0x7fb0e813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:21.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.294+0000 7fb0f0025640 10 monclient(hunting): picked mon.noname-b con 0x7fb0e81430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:21.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.294+0000 7fb0f0025640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e80ab9c0 con 0x7fb0e813f710 2024-03-25T18:09:21.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.294+0000 7fb0f0025640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e80abca0 con 0x7fb0e81430c0 2024-03-25T18:09:21.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.294+0000 7fb0f0025640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb0e8137e80 con 0x7fb0e80a98f0 2024-03-25T18:09:21.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.294+0000 7fb0f0025640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:21.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.294+0000 7fb0f0025640 10 monclient(hunting): authenticate will time out at 6916.546957s 2024-03-25T18:09:21.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.300+0000 7fb0ee5c2640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fb0e81430c0 0x7fb0e81e2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:47176/0 (socket says 172.21.15.37:47176) 2024-03-25T18:09:21.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.300+0000 7fb0ee5c2640 1 -- 172.21.15.37:0/76829113 learned_addr learned my addr 172.21.15.37:0/76829113 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:21.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.302+0000 7fb0de7fc640 10 client.?.objecter ms_handle_connect 0x7fb0e81430c0 2024-03-25T18:09:21.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.302+0000 7fb0de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:21.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.303+0000 7fb0de7fc640 10 client.?.objecter ms_handle_connect 0x7fb0e80a98f0 2024-03-25T18:09:21.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.303+0000 7fb0de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:21.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.307+0000 7fb0de7fc640 10 client.?.objecter ms_handle_connect 0x7fb0e813f710 2024-03-25T18:09:21.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.307+0000 7fb0de7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:21.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.309+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044524357 0 0) 0x7fb0e80abca0 con 0x7fb0e81430c0 2024-03-25T18:09:21.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.309+0000 7fb0de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:21.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.309+0000 7fb0de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:21.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.309+0000 7fb0de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:21.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.309+0000 7fb0de7fc640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:09:21.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.309+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0cc004eb0 con 0x7fb0e81430c0 2024-03-25T18:09:21.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.311+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327644551 0 0) 0x7fb0e80ab9c0 con 0x7fb0e813f710 2024-03-25T18:09:21.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.311+0000 7fb0de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:21.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.311+0000 7fb0de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:21.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.311+0000 7fb0de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:21.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.311+0000 7fb0de7fc640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:09:21.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.311+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e80abca0 con 0x7fb0e813f710 2024-03-25T18:09:21.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.323+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 402678955 0 0) 0x7fb0e8137e80 con 0x7fb0e80a98f0 2024-03-25T18:09:21.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.323+0000 7fb0de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:21.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.323+0000 7fb0de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:21.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.323+0000 7fb0de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:21.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.323+0000 7fb0de7fc640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:09:21.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.323+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb0e80ab9c0 con 0x7fb0e80a98f0 2024-03-25T18:09:21.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.324+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1015036161 0 0) 0x7fb0cc004eb0 con 0x7fb0e81430c0 2024-03-25T18:09:21.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.324+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0e8137e80 con 0x7fb0e81430c0 2024-03-25T18:09:21.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.324+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb0e0001c30 con 0x7fb0e81430c0 2024-03-25T18:09:21.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.324+0000 7fb0de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:21.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.324+0000 7fb0de7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:21.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.324+0000 7fb0de7fc640 10 monclient(hunting): dump: 2024-03-25T18:09:21.325 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:21.326 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:21.326 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:21.326 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:21.326 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:21.326 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:21.326 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:21.326 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:21.326 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:21.326 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:21.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.324+0000 7fb0de7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:21.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.330+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 861663370 0 0) 0x7fb0e80abca0 con 0x7fb0e813f710 2024-03-25T18:09:21.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.330+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb0cc004eb0 con 0x7fb0e813f710 2024-03-25T18:09:21.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.330+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb0d8001ae0 con 0x7fb0e813f710 2024-03-25T18:09:21.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.330+0000 7fb0de7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.330+0000 7fb0de7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.330+0000 7fb0de7fc640 10 monclient(hunting): dump: 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:21.332 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:21.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2287590307 0 0) 0x7fb0cc004eb0 con 0x7fb0e813f710 2024-03-25T18:09:21.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 >> v1:172.21.15.37:6790/0 conn(0x7fb0e80a98f0 legacy=0x7fb0e81d7a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:21.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 >> v1:172.21.15.117:6789/0 conn(0x7fb0e81430c0 legacy=0x7fb0e81e2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0de7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0de7fc640 1 monclient: found mon.a 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0de7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e81d8150 con 0x7fb0e813f710 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0de7fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0de7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0f0025640 5 monclient: authenticate success, global_id 4290 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0f0025640 10 monclient: _renew_subs 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0f0025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:09:21.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.340+0000 7fb0f0025640 1 -- 172.21.15.37:0/76829113 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e8138eb0 con 0x7fb0e813f710 2024-03-25T18:09:21.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.341+0000 7fb0f0025640 10 client.4290.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:21.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.341+0000 7fb0f0025640 10 monclient: _renew_subs 2024-03-25T18:09:21.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.341+0000 7fb0f0025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:09:21.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.341+0000 7fb0f0025640 1 -- 172.21.15.37:0/76829113 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e8137b20 con 0x7fb0e813f710 2024-03-25T18:09:21.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.341+0000 7fb0f0025640 1 librados: init done 2024-03-25T18:09:21.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.342+0000 7fb0f0025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:21.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.342+0000 7fb0f0025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:21.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.342+0000 7fb0f0025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:09:21.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.342+0000 7fb0f0025640 1 -- 172.21.15.37:0/76829113 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e80627b0 con 0x7fb0e813f710 2024-03-25T18:09:21.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.372+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb0d8001210 con 0x7fb0e813f710 2024-03-25T18:09:21.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.372+0000 7fb0de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:21.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.372+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb0d8004c30 con 0x7fb0e813f710 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.372+0000 7fb0de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.373+0000 7fb0de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.373+0000 7fb0de7fc640 10 monclient: dump: 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:21.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.373+0000 7fb0dffff640 4 set_mon_vals no callback set 2024-03-25T18:09:21.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.373+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fb0d801ac00 con 0x7fb0e813f710 2024-03-25T18:09:21.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.374+0000 7fb0eddc1640 10 monclient: get_auth_request con 0x7fb0cc06dfb0 auth_method 0 2024-03-25T18:09:21.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.374+0000 7fb0de7fc640 10 client.4290.objecter ms_handle_connect 0x7fb0cc06dfb0 2024-03-25T18:09:21.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.377+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fb0d8054900 con 0x7fb0e813f710 2024-03-25T18:09:21.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.377+0000 7fb0de7fc640 10 client.4290.objecter ms_dispatch 0x7fb0e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:21.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.377+0000 7fb0de7fc640 3 client.4290.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:21.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.377+0000 7fb0de7fc640 3 client.4290.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:21.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.377+0000 7fb0de7fc640 20 client.4290.objecter dump_active .. 0 homeless 2024-03-25T18:09:21.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.678+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb0d8050240 con 0x7fb0e813f710 2024-03-25T18:09:21.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.678+0000 7fb0de7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:21.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.678+0000 7fb0de7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:21.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.902+0000 7fb0f0025640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:21.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.902+0000 7fb0f0025640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:21.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.902+0000 7fb0f0025640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:09:21.903 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.902+0000 7fb0f0025640 1 -- 172.21.15.37:0/76829113 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0e80b71a0 con 0x7fb0e813f710 2024-03-25T18:09:21.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.927+0000 7fb0de7fc640 1 -- 172.21.15.37:0/76829113 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fb0d8052340 con 0x7fb0e813f710 2024-03-25T18:09:21.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.927+0000 7fb0de7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:21.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.927+0000 7fb0de7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:21.929 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:09:21.929 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:21.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.929+0000 7fb0f0025640 10 librados: watch_flush enter 2024-03-25T18:09:21.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 10 librados: watch_flush exit 2024-03-25T18:09:21.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 20 client.4290.objecter shutdown clearing up homeless session... 2024-03-25T18:09:21.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 10 client.4290.objecter successfully canceled tick 2024-03-25T18:09:21.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 1 -- 172.21.15.37:0/76829113 >> v1:172.21.15.117:6800/3703993921 conn(0x7fb0cc06dfb0 legacy=0x7fb0cc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:21.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 10 monclient: shutdown 2024-03-25T18:09:21.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:21.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 1 -- 172.21.15.37:0/76829113 >> v1:172.21.15.37:6789/0 conn(0x7fb0e813f710 legacy=0x7fb0e8133fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:21.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 1 -- 172.21.15.37:0/76829113 shutdown_connections 2024-03-25T18:09:21.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 1 -- 172.21.15.37:0/76829113 >> 172.21.15.37:0/76829113 conn(0x7fb0e80b7ea0 msgr2=0x7fb0e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:21.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 1 -- 172.21.15.37:0/76829113 shutdown_connections 2024-03-25T18:09:21.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 1 -- 172.21.15.37:0/76829113 wait complete. 2024-03-25T18:09:21.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:21.930+0000 7fb0f0025640 1 librados: shutdown 2024-03-25T18:09:21.940 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:28.747 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:28.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.861+0000 7ff74875d640 10 monclient: get_monmap_and_config 2024-03-25T18:09:28.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.861+0000 7ff74875d640 10 monclient: build_initial_monmap 2024-03-25T18:09:28.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.861+0000 7ff74875d640 10 monclient: monmap: 2024-03-25T18:09:28.863 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:28.863 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:28.863 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:09:28.862589+0000 2024-03-25T18:09:28.863 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:09:28.862589+0000 2024-03-25T18:09:28.863 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:28.863 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:28.863 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:28.864 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:09:28.864 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:09:28.864 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:28.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.862+0000 7ff74875d640 1 Processor -- start 2024-03-25T18:09:28.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.862+0000 7ff74875d640 1 -- start start 2024-03-25T18:09:28.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.862+0000 7ff74875d640 10 monclient: init 2024-03-25T18:09:28.865 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.863+0000 7ff74875d640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:28.865 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.863+0000 7ff74875d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:28.865 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.863+0000 7ff74875d640 10 monclient(hunting): picked mon.noname-a con 0x7ff7400b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:28.865 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.863+0000 7ff74875d640 10 monclient(hunting): picked mon.noname-b con 0x7ff7400b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:28.865 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.863+0000 7ff74875d640 10 monclient(hunting): picked mon.noname-c con 0x7ff74014be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:28.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.863+0000 7ff74875d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7400af940 con 0x7ff7400b4d60 2024-03-25T18:09:28.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.863+0000 7ff74875d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7400b0a70 con 0x7ff74014be90 2024-03-25T18:09:28.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.863+0000 7ff74875d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff7400b0d50 con 0x7ff7400b1730 2024-03-25T18:09:28.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.863+0000 7ff74875d640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:28.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.863+0000 7ff74875d640 10 monclient(hunting): authenticate will time out at 6924.115975s 2024-03-25T18:09:28.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.877+0000 7ff745cf8640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7ff7400b1730 0x7ff7400b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:48762/0 (socket says 172.21.15.37:48762) 2024-03-25T18:09:28.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.877+0000 7ff745cf8640 1 -- 172.21.15.37:0/436319886 learned_addr learned my addr 172.21.15.37:0/436319886 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.885+0000 7ff7454f7640 1 -- 172.21.15.37:0/436319886 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 788988072 0 0) 0x7ff7400b0d50 con 0x7ff7400b1730 2024-03-25T18:09:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.885+0000 7ff7454f7640 10 monclient(hunting): _negotiate 2024-03-25T18:09:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.885+0000 7ff7454f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.885+0000 7ff7454f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.885+0000 7ff7454f7640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:09:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.885+0000 7ff7454f7640 1 -- 172.21.15.37:0/436319886 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff72c003520 con 0x7ff7400b1730 2024-03-25T18:09:28.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.893+0000 7ff7454f7640 1 -- 172.21.15.37:0/436319886 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3506056195 0 0) 0x7ff7400b0a70 con 0x7ff74014be90 2024-03-25T18:09:28.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.893+0000 7ff7454f7640 10 monclient(hunting): _negotiate 2024-03-25T18:09:28.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.893+0000 7ff7454f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:28.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.893+0000 7ff7454f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:28.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.893+0000 7ff7454f7640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:09:28.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.893+0000 7ff7454f7640 1 -- 172.21.15.37:0/436319886 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff7400b0d50 con 0x7ff74014be90 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 1 -- 172.21.15.37:0/436319886 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570480837 0 0) 0x7ff72c003520 con 0x7ff7400b1730 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 1 -- 172.21.15.37:0/436319886 >> v1:172.21.15.117:6789/0 conn(0x7ff74014be90 legacy=0x7ff74014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 1 -- 172.21.15.37:0/436319886 >> v1:172.21.15.37:6789/0 conn(0x7ff7400b4d60 legacy=0x7ff74013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 10 monclient: _finish_hunting 0 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 1 monclient: found mon.noname-b 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 1 -- 172.21.15.37:0/436319886 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7400b1030 con 0x7ff7400b1730 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 10 monclient: _finish_auth 0 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 10 monclient: _check_auth_tickets 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 1 -- 172.21.15.37:0/436319886 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7ff730001690 con 0x7ff7400b1730 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:28.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff7454f7640 10 monclient: dump: 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff74875d640 5 monclient: authenticate success, global_id 4292 2024-03-25T18:09:28.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.899+0000 7ff74875d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:28.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.927+0000 7ff7454f7640 1 -- 172.21.15.37:0/436319886 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff7300017f0 con 0x7ff7400b1730 2024-03-25T18:09:28.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.927+0000 7ff7454f7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:28.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.927+0000 7ff7454f7640 1 -- 172.21.15.37:0/436319886 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7ff730001c00 con 0x7ff7400b1730 2024-03-25T18:09:28.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.927+0000 7ff74875d640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:28.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.927+0000 7ff74875d640 4 set_mon_vals no callback set 2024-03-25T18:09:28.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.927+0000 7ff7454f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:28.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.927+0000 7ff7454f7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:28.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.927+0000 7ff7454f7640 10 monclient: dump: 2024-03-25T18:09:28.928 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:28.928 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:28.929 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:28.929 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:28.929 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:28.929 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:28.929 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:28.929 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:28.929 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:28.929 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.928+0000 7ff74875d640 10 monclient: shutdown 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.928+0000 7ff74875d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.928+0000 7ff74875d640 1 -- 172.21.15.37:0/436319886 >> v1:172.21.15.37:6790/0 conn(0x7ff7400b1730 legacy=0x7ff7400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.928+0000 7ff74875d640 1 -- 172.21.15.37:0/436319886 shutdown_connections 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.928+0000 7ff74875d640 1 -- 172.21.15.37:0/436319886 >> 172.21.15.37:0/436319886 conn(0x7ff7400617a0 msgr2=0x7ff740061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.928+0000 7ff74875d640 1 -- 172.21.15.37:0/436319886 shutdown_connections 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.928+0000 7ff74875d640 1 -- 172.21.15.37:0/436319886 wait complete. 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.929+0000 7ff74875d640 10 monclient: build_initial_monmap 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.929+0000 7ff74875d640 10 monclient: monmap: 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:09:28.932 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.929+0000 7ff74875d640 1 librados: starting msgr at 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.929+0000 7ff74875d640 1 librados: starting objecter 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.929+0000 7ff74875d640 1 Processor -- start 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.929+0000 7ff74875d640 1 -- start start 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.929+0000 7ff74875d640 1 librados: setting wanted keys 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.929+0000 7ff74875d640 1 librados: calling monclient init 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.929+0000 7ff74875d640 10 monclient: init 2024-03-25T18:09:28.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.930+0000 7ff74875d640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:28.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.930+0000 7ff74875d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:28.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.930+0000 7ff74875d640 10 monclient(hunting): picked mon.noname-b con 0x7ff7400b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:28.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.930+0000 7ff74875d640 10 monclient(hunting): picked mon.noname-a con 0x7ff7400b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:28.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.930+0000 7ff74875d640 10 monclient(hunting): picked mon.noname-c con 0x7ff74014be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:28.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.930+0000 7ff74875d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74013b810 con 0x7ff7400b4d60 2024-03-25T18:09:28.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.930+0000 7ff74875d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff74013a730 con 0x7ff7400b1730 2024-03-25T18:09:28.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.930+0000 7ff74875d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff740139140 con 0x7ff74014be90 2024-03-25T18:09:28.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.930+0000 7ff74875d640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:28.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.930+0000 7ff74875d640 10 monclient(hunting): authenticate will time out at 6924.183214s 2024-03-25T18:09:28.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.950+0000 7ff7464f9640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7ff7400b1730 0x7ff740139c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:59994/0 (socket says 172.21.15.37:59994) 2024-03-25T18:09:28.951 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.950+0000 7ff7464f9640 1 -- 172.21.15.37:0/565010586 learned_addr learned my addr 172.21.15.37:0/565010586 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:28.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.954+0000 7ff736ffd640 10 client.?.objecter ms_handle_connect 0x7ff7400b1730 2024-03-25T18:09:28.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.954+0000 7ff736ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:28.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.955+0000 7ff736ffd640 10 client.?.objecter ms_handle_connect 0x7ff74014be90 2024-03-25T18:09:28.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.955+0000 7ff736ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:28.958 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.957+0000 7ff736ffd640 10 client.?.objecter ms_handle_connect 0x7ff7400b4d60 2024-03-25T18:09:28.958 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.957+0000 7ff736ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:28.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.960+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691898551 0 0) 0x7ff74013b810 con 0x7ff7400b4d60 2024-03-25T18:09:28.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.960+0000 7ff736ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:28.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.960+0000 7ff736ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:28.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.960+0000 7ff736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:28.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.960+0000 7ff736ffd640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:09:28.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.960+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff724004eb0 con 0x7ff7400b4d60 2024-03-25T18:09:28.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.962+0000 7ff745cf8640 1 -- 172.21.15.37:0/565010586 >> v1:172.21.15.37:6789/0 conn(0x7ff7400b4d60 legacy=0x7ff74012fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-03-25T18:09:28.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.962+0000 7ff745cf8640 1 -- 172.21.15.37:0/565010586 >> v1:172.21.15.37:6789/0 conn(0x7ff7400b4d60 legacy=0x7ff74012fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:09:28.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.962+0000 7ff745cf8640 1 --1- 172.21.15.37:0/565010586 >> v1:172.21.15.37:6789/0 conn(0x7ff7400b4d60 0x7ff74012fef0 :-1 s=OPENED pgs=8503 cs=1 l=1).handle_message read tag failed 2024-03-25T18:09:28.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.962+0000 7ff745cf8640 1 --1- 172.21.15.37:0/565010586 >> v1:172.21.15.37:6789/0 conn(0x7ff7400b4d60 0x7ff74012fef0 :-1 s=OPENED pgs=8503 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:09:28.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.962+0000 7ff736ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.37:6789/0 2024-03-25T18:09:28.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.962+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2353351768 0 0) 0x7ff74013a730 con 0x7ff7400b1730 2024-03-25T18:09:28.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.962+0000 7ff736ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:28.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.962+0000 7ff736ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:28.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.962+0000 7ff736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:28.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.962+0000 7ff736ffd640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:09:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.962+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff74013b810 con 0x7ff7400b1730 2024-03-25T18:09:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.963+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1597964357 0 0) 0x7ff740139140 con 0x7ff74014be90 2024-03-25T18:09:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.963+0000 7ff736ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.963+0000 7ff736ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.963+0000 7ff736ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.963+0000 7ff736ffd640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:09:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.963+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff74013a730 con 0x7ff74014be90 2024-03-25T18:09:28.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.972+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 162886103 0 0) 0x7ff74013a730 con 0x7ff74014be90 2024-03-25T18:09:28.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.972+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff740139140 con 0x7ff74014be90 2024-03-25T18:09:28.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.972+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7ff738001bd0 con 0x7ff74014be90 2024-03-25T18:09:28.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.972+0000 7ff736ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:28.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.972+0000 7ff736ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:28.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.972+0000 7ff736ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:28.973 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:28.973 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:28.973 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:28.973 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:28.974 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:28.974 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:28.974 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:28.974 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:28.974 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:28.974 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:28.974 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.972+0000 7ff736ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:28.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.993+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2335163971 0 0) 0x7ff74013b810 con 0x7ff7400b1730 2024-03-25T18:09:28.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.993+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff74013a730 con 0x7ff7400b1730 2024-03-25T18:09:28.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.993+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7ff73c001930 con 0x7ff7400b1730 2024-03-25T18:09:28.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.993+0000 7ff736ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:28.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.993+0000 7ff736ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:28.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:28.993+0000 7ff736ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:28.994 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:28.994 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:28.994 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:28.995 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:28.995 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:28.995 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:28.995 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:28.995 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:28.995 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:28.995 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1240506022 0 0) 0x7ff740139140 con 0x7ff74014be90 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 >> v1:172.21.15.117:6789/0 conn(0x7ff7400b1730 legacy=0x7ff740139c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 >> v1:172.21.15.37:6789/0 conn(0x7ff7400b4d60 legacy=0x7ff74012fef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff736ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff736ffd640 1 monclient: found mon.c 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff736ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff736ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74013bbc0 con 0x7ff74014be90 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff736ffd640 10 monclient: _finish_auth 0 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff736ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff736ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff74875d640 5 monclient: authenticate success, global_id 4295 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff74875d640 10 monclient: _renew_subs 2024-03-25T18:09:29.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff74875d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:29.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.005+0000 7ff74875d640 1 -- 172.21.15.37:0/565010586 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74013a330 con 0x7ff74014be90 2024-03-25T18:09:29.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.006+0000 7ff74875d640 10 client.4295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:29.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.006+0000 7ff74875d640 10 monclient: _renew_subs 2024-03-25T18:09:29.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.006+0000 7ff74875d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:29.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.006+0000 7ff74875d640 1 -- 172.21.15.37:0/565010586 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7401307e0 con 0x7ff74014be90 2024-03-25T18:09:29.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.006+0000 7ff74875d640 1 librados: init done 2024-03-25T18:09:29.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.007+0000 7ff74875d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:29.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.007+0000 7ff74875d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:29.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.007+0000 7ff74875d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:29.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.007+0000 7ff74875d640 1 -- 172.21.15.37:0/565010586 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7400627b0 con 0x7ff74014be90 2024-03-25T18:09:29.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.330+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff738001fe0 con 0x7ff74014be90 2024-03-25T18:09:29.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.330+0000 7ff736ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:29.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.330+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7ff738003b10 con 0x7ff74014be90 2024-03-25T18:09:29.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.330+0000 7ff736ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:29.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.330+0000 7ff736ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:29.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.330+0000 7ff736ffd640 10 monclient: dump: 2024-03-25T18:09:29.334 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:29.334 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:29.334 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:29.334 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:29.334 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:29.335 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:29.335 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:29.335 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:29.335 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:29.335 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:29.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.330+0000 7ff744cf6640 4 set_mon_vals no callback set 2024-03-25T18:09:29.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.331+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7ff738019ae0 con 0x7ff74014be90 2024-03-25T18:09:29.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.332+0000 7ff7464f9640 10 monclient: get_auth_request con 0x7ff72406df30 auth_method 0 2024-03-25T18:09:29.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.332+0000 7ff736ffd640 10 client.4295.objecter ms_handle_connect 0x7ff72406df30 2024-03-25T18:09:29.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.339+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7ff738053650 con 0x7ff74014be90 2024-03-25T18:09:29.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.339+0000 7ff736ffd640 10 client.4295.objecter ms_dispatch 0x7ff740000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:29.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.339+0000 7ff736ffd640 3 client.4295.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:29.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.339+0000 7ff736ffd640 3 client.4295.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:29.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.339+0000 7ff736ffd640 20 client.4295.objecter dump_active .. 0 homeless 2024-03-25T18:09:29.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.339+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff73804f160 con 0x7ff74014be90 2024-03-25T18:09:29.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.339+0000 7ff736ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:29.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.339+0000 7ff736ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:29.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.582+0000 7ff74875d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:29.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.582+0000 7ff74875d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:29.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.582+0000 7ff74875d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:29.583 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.582+0000 7ff74875d640 1 -- 172.21.15.37:0/565010586 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff74013bd50 con 0x7ff74014be90 2024-03-25T18:09:29.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.640+0000 7ff736ffd640 1 -- 172.21.15.37:0/565010586 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7ff738051090 con 0x7ff74014be90 2024-03-25T18:09:29.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.640+0000 7ff736ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:29.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.640+0000 7ff736ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:29.642 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:09:29.642 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:29.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.643+0000 7ff74875d640 10 librados: watch_flush enter 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.643+0000 7ff74875d640 10 librados: watch_flush exit 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.643+0000 7ff74875d640 20 client.4295.objecter shutdown clearing up homeless session... 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.643+0000 7ff74875d640 10 client.4295.objecter successfully canceled tick 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.643+0000 7ff74875d640 1 -- 172.21.15.37:0/565010586 >> v1:172.21.15.117:6800/3703993921 conn(0x7ff72406df30 legacy=0x7ff7240703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.643+0000 7ff74875d640 10 monclient: shutdown 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.643+0000 7ff74875d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.643+0000 7ff74875d640 1 -- 172.21.15.37:0/565010586 >> v1:172.21.15.37:6790/0 conn(0x7ff74014be90 legacy=0x7ff7401d7a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.644+0000 7ff74875d640 1 -- 172.21.15.37:0/565010586 shutdown_connections 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.644+0000 7ff74875d640 1 -- 172.21.15.37:0/565010586 >> 172.21.15.37:0/565010586 conn(0x7ff7400617a0 msgr2=0x7ff740061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.644+0000 7ff74875d640 1 -- 172.21.15.37:0/565010586 shutdown_connections 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.644+0000 7ff74875d640 1 -- 172.21.15.37:0/565010586 wait complete. 2024-03-25T18:09:29.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:29.644+0000 7ff74875d640 1 librados: shutdown 2024-03-25T18:09:29.656 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:36.462 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.577+0000 7fce3af2b640 10 monclient: get_monmap_and_config 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.577+0000 7fce3af2b640 10 monclient: build_initial_monmap 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.577+0000 7fce3af2b640 10 monclient: monmap: 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:09:36.578183+0000 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:09:36.578183+0000 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:09:36.578 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:09:36.579 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:36.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.578+0000 7fce3af2b640 1 Processor -- start 2024-03-25T18:09:36.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.578+0000 7fce3af2b640 1 -- start start 2024-03-25T18:09:36.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.578+0000 7fce3af2b640 10 monclient: init 2024-03-25T18:09:36.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.578+0000 7fce3af2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:36.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.578+0000 7fce3af2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:09:36.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.579+0000 7fce3af2b640 10 monclient(hunting): picked mon.noname-a con 0x7fce340b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:36.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.579+0000 7fce3af2b640 10 monclient(hunting): picked mon.noname-b con 0x7fce340b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:36.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.579+0000 7fce3af2b640 10 monclient(hunting): picked mon.noname-c con 0x7fce3414be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:36.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.579+0000 7fce3af2b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce340af940 con 0x7fce340b4d60 2024-03-25T18:09:36.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.579+0000 7fce3af2b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce340b0a70 con 0x7fce3414be90 2024-03-25T18:09:36.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.579+0000 7fce3af2b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce340b0d50 con 0x7fce340b1730 2024-03-25T18:09:36.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.579+0000 7fce3af2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:36.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.579+0000 7fce3af2b640 10 monclient(hunting): authenticate will time out at 6931.831584s 2024-03-25T18:09:36.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.585+0000 7fce33fff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fce340b1730 0x7fce340b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:53494/0 (socket says 172.21.15.37:53494) 2024-03-25T18:09:36.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.585+0000 7fce33fff640 1 -- 172.21.15.37:0/3505075923 learned_addr learned my addr 172.21.15.37:0/3505075923 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:36.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.600+0000 7fce337fe640 1 -- 172.21.15.37:0/3505075923 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 757969491 0 0) 0x7fce340b0d50 con 0x7fce340b1730 2024-03-25T18:09:36.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.600+0000 7fce337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.600+0000 7fce337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.600+0000 7fce337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.600+0000 7fce337fe640 10 monclient(hunting): my global_id is 4298 2024-03-25T18:09:36.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.600+0000 7fce337fe640 1 -- 172.21.15.37:0/3505075923 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce24003520 con 0x7fce340b1730 2024-03-25T18:09:36.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.600+0000 7fce337fe640 1 -- 172.21.15.37:0/3505075923 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1063343778 0 0) 0x7fce340b0a70 con 0x7fce3414be90 2024-03-25T18:09:36.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.600+0000 7fce337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.600+0000 7fce337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.600+0000 7fce337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.600+0000 7fce337fe640 10 monclient(hunting): my global_id is 4303 2024-03-25T18:09:36.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.601+0000 7fce337fe640 1 -- 172.21.15.37:0/3505075923 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce340b0d50 con 0x7fce3414be90 2024-03-25T18:09:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 1 -- 172.21.15.37:0/3505075923 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1017122062 0 0) 0x7fce340b0d50 con 0x7fce3414be90 2024-03-25T18:09:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 1 -- 172.21.15.37:0/3505075923 >> v1:172.21.15.37:6790/0 conn(0x7fce340b1730 legacy=0x7fce340b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 1 -- 172.21.15.37:0/3505075923 >> v1:172.21.15.37:6789/0 conn(0x7fce340b4d60 legacy=0x7fce3413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 10 monclient: _finish_hunting 0 2024-03-25T18:09:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 1 monclient: found mon.noname-c 2024-03-25T18:09:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:09:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 1 -- 172.21.15.37:0/3505075923 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce340b1030 con 0x7fce3414be90 2024-03-25T18:09:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 10 monclient: _finish_auth 0 2024-03-25T18:09:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 10 monclient: _check_auth_tickets 2024-03-25T18:09:36.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 1 -- 172.21.15.37:0/3505075923 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fce2c0019f0 con 0x7fce3414be90 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce337fe640 10 monclient: dump: 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:36.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce3af2b640 5 monclient: authenticate success, global_id 4303 2024-03-25T18:09:36.613 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.610+0000 7fce3af2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:36.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.638+0000 7fce337fe640 1 -- 172.21.15.37:0/3505075923 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce2c001b50 con 0x7fce3414be90 2024-03-25T18:09:36.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.638+0000 7fce337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:36.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.638+0000 7fce337fe640 1 -- 172.21.15.37:0/3505075923 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fce2c001f60 con 0x7fce3414be90 2024-03-25T18:09:36.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.638+0000 7fce3af2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:36.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.638+0000 7fce3af2b640 4 set_mon_vals no callback set 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.638+0000 7fce337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.638+0000 7fce337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.638+0000 7fce337fe640 10 monclient: dump: 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:36.640 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:36.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.639+0000 7fce3af2b640 10 monclient: shutdown 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.639+0000 7fce3af2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.639+0000 7fce3af2b640 1 -- 172.21.15.37:0/3505075923 >> v1:172.21.15.117:6789/0 conn(0x7fce3414be90 legacy=0x7fce3414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.639+0000 7fce3af2b640 1 -- 172.21.15.37:0/3505075923 shutdown_connections 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.639+0000 7fce3af2b640 1 -- 172.21.15.37:0/3505075923 >> 172.21.15.37:0/3505075923 conn(0x7fce340617a0 msgr2=0x7fce34061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.639+0000 7fce3af2b640 1 -- 172.21.15.37:0/3505075923 shutdown_connections 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.639+0000 7fce3af2b640 1 -- 172.21.15.37:0/3505075923 wait complete. 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.640+0000 7fce3af2b640 10 monclient: build_initial_monmap 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.640+0000 7fce3af2b640 10 monclient: monmap: 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:09:36.642 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.641+0000 7fce3af2b640 1 librados: starting msgr at 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.641+0000 7fce3af2b640 1 librados: starting objecter 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.641+0000 7fce3af2b640 1 Processor -- start 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.641+0000 7fce3af2b640 1 -- start start 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.641+0000 7fce3af2b640 1 librados: setting wanted keys 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.641+0000 7fce3af2b640 1 librados: calling monclient init 2024-03-25T18:09:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.641+0000 7fce3af2b640 10 monclient: init 2024-03-25T18:09:36.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.642+0000 7fce3af2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:36.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.642+0000 7fce3af2b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:36.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.642+0000 7fce3af2b640 10 monclient(hunting): picked mon.noname-b con 0x7fce340b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:36.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.642+0000 7fce3af2b640 10 monclient(hunting): picked mon.noname-c con 0x7fce340b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:36.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.642+0000 7fce3af2b640 10 monclient(hunting): picked mon.noname-a con 0x7fce3414be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:36.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.642+0000 7fce3af2b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce341db9b0 con 0x7fce3414be90 2024-03-25T18:09:36.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.642+0000 7fce3af2b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce340a9650 con 0x7fce340b1730 2024-03-25T18:09:36.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.642+0000 7fce3af2b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce34142680 con 0x7fce340b4d60 2024-03-25T18:09:36.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.642+0000 7fce3af2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:36.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.642+0000 7fce3af2b640 10 monclient(hunting): authenticate will time out at 6931.894883s 2024-03-25T18:09:36.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.660+0000 7fce38cc7640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fce340b1730 0x7fce34143100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:36374/0 (socket says 172.21.15.37:36374) 2024-03-25T18:09:36.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.660+0000 7fce38cc7640 1 -- 172.21.15.37:0/3827650469 learned_addr learned my addr 172.21.15.37:0/3827650469 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:36.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.664+0000 7fce317fa640 10 client.?.objecter ms_handle_connect 0x7fce340b4d60 2024-03-25T18:09:36.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.664+0000 7fce317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:36.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.665+0000 7fce317fa640 10 client.?.objecter ms_handle_connect 0x7fce340b1730 2024-03-25T18:09:36.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.665+0000 7fce317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:36.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.677+0000 7fce317fa640 10 client.?.objecter ms_handle_connect 0x7fce3414be90 2024-03-25T18:09:36.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.677+0000 7fce317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:36.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.677+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088766482 0 0) 0x7fce34142680 con 0x7fce340b4d60 2024-03-25T18:09:36.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.677+0000 7fce317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.677+0000 7fce317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.677+0000 7fce317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.677+0000 7fce317fa640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:09:36.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.677+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce18004eb0 con 0x7fce340b4d60 2024-03-25T18:09:36.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.678+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1146100736 0 0) 0x7fce340a9650 con 0x7fce340b1730 2024-03-25T18:09:36.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.678+0000 7fce317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.678+0000 7fce317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.678+0000 7fce317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.678+0000 7fce317fa640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:09:36.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.678+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce34142680 con 0x7fce340b1730 2024-03-25T18:09:36.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.685+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861206110 0 0) 0x7fce341db9b0 con 0x7fce3414be90 2024-03-25T18:09:36.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.685+0000 7fce317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:09:36.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.685+0000 7fce317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:36.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.685+0000 7fce317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:36.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.685+0000 7fce317fa640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:09:36.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.685+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce340a9650 con 0x7fce3414be90 2024-03-25T18:09:36.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.694+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4137055642 0 0) 0x7fce18004eb0 con 0x7fce340b4d60 2024-03-25T18:09:36.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.694+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce341db9b0 con 0x7fce340b4d60 2024-03-25T18:09:36.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.694+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fce20001950 con 0x7fce340b4d60 2024-03-25T18:09:36.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.694+0000 7fce317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.694+0000 7fce317fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:36.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.694+0000 7fce317fa640 10 monclient(hunting): dump: 2024-03-25T18:09:36.696 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:36.696 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:36.696 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:36.696 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:36.696 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.696 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:36.696 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:36.696 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:36.696 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:36.696 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:36.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.694+0000 7fce317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:36.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.698+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4244278360 0 0) 0x7fce34142680 con 0x7fce340b1730 2024-03-25T18:09:36.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.698+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce18004eb0 con 0x7fce340b1730 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.698+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fce280019b0 con 0x7fce340b1730 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.698+0000 7fce317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.698+0000 7fce317fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.698+0000 7fce317fa640 10 monclient(hunting): dump: 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:36.699 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:36.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.701+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3428102933 0 0) 0x7fce340a9650 con 0x7fce3414be90 2024-03-25T18:09:36.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.701+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce34142680 con 0x7fce3414be90 2024-03-25T18:09:36.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.701+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fce2c001e60 con 0x7fce3414be90 2024-03-25T18:09:36.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.701+0000 7fce317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:36.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.701+0000 7fce317fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:36.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.701+0000 7fce317fa640 10 monclient(hunting): dump: 2024-03-25T18:09:36.703 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:36.703 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:36.703 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:36.703 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:36.703 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:36.703 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:36.703 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:36.703 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:36.703 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:36.703 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:36.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 471511277 0 0) 0x7fce341db9b0 con 0x7fce340b4d60 2024-03-25T18:09:36.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 >> v1:172.21.15.117:6789/0 conn(0x7fce340b1730 legacy=0x7fce34143100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 >> v1:172.21.15.37:6789/0 conn(0x7fce3414be90 legacy=0x7fce34137e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:36.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce317fa640 10 monclient: _finish_hunting 0 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce317fa640 1 monclient: found mon.c 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce317fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce34142810 con 0x7fce340b4d60 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce317fa640 10 monclient: _finish_auth 0 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce317fa640 10 monclient: _check_auth_tickets 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce3af2b640 5 monclient: authenticate success, global_id 4301 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce3af2b640 10 monclient: _renew_subs 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce3af2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce3af2b640 1 -- 172.21.15.37:0/3827650469 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce34142a40 con 0x7fce340b4d60 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce3af2b640 10 client.4301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce3af2b640 10 monclient: _renew_subs 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce3af2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce3af2b640 1 -- 172.21.15.37:0/3827650469 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce34140560 con 0x7fce340b4d60 2024-03-25T18:09:36.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.706+0000 7fce3af2b640 1 librados: init done 2024-03-25T18:09:36.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.707+0000 7fce3af2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:36.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.707+0000 7fce3af2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:36.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.707+0000 7fce3af2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:36.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:36.707+0000 7fce3af2b640 1 -- 172.21.15.37:0/3827650469 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce340a9470 con 0x7fce340b4d60 2024-03-25T18:09:37.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.016+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce20001d60 con 0x7fce340b4d60 2024-03-25T18:09:37.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.016+0000 7fce317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:37.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.016+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fce20004980 con 0x7fce340b4d60 2024-03-25T18:09:37.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.016+0000 7fce317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:37.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.016+0000 7fce317fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:37.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.016+0000 7fce317fa640 10 monclient: dump: 2024-03-25T18:09:37.017 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:37.017 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:37.018 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:37.018 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:37.018 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:37.018 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:37.018 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:37.018 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:37.018 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:37.018 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:37.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.016+0000 7fce32ffd640 4 set_mon_vals no callback set 2024-03-25T18:09:37.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.017+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fce2001a9d0 con 0x7fce340b4d60 2024-03-25T18:09:37.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.017+0000 7fce38cc7640 10 monclient: get_auth_request con 0x7fce1806e0b0 auth_method 0 2024-03-25T18:09:37.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.018+0000 7fce317fa640 10 client.4301.objecter ms_handle_connect 0x7fce1806e0b0 2024-03-25T18:09:37.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.024+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fce20054780 con 0x7fce340b4d60 2024-03-25T18:09:37.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.024+0000 7fce317fa640 10 client.4301.objecter ms_dispatch 0x7fce34000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:37.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.024+0000 7fce317fa640 3 client.4301.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:37.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.024+0000 7fce317fa640 3 client.4301.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:37.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.024+0000 7fce317fa640 20 client.4301.objecter dump_active .. 0 homeless 2024-03-25T18:09:37.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.024+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce200500c0 con 0x7fce340b4d60 2024-03-25T18:09:37.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.024+0000 7fce317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:37.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.024+0000 7fce317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:37.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.252+0000 7fce3af2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:37.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.252+0000 7fce3af2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:37.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.252+0000 7fce3af2b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:37.253 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.252+0000 7fce3af2b640 1 -- 172.21.15.37:0/3827650469 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce340627b0 con 0x7fce340b4d60 2024-03-25T18:09:37.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.279+0000 7fce317fa640 1 -- 172.21.15.37:0/3827650469 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fce200521c0 con 0x7fce340b4d60 2024-03-25T18:09:37.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.279+0000 7fce317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:37.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.279+0000 7fce317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:37.281 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:09:37.281 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:37.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.281+0000 7fce3af2b640 10 librados: watch_flush enter 2024-03-25T18:09:37.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.281+0000 7fce3af2b640 10 librados: watch_flush exit 2024-03-25T18:09:37.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.281+0000 7fce3af2b640 20 client.4301.objecter shutdown clearing up homeless session... 2024-03-25T18:09:37.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.281+0000 7fce3af2b640 10 client.4301.objecter successfully canceled tick 2024-03-25T18:09:37.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.282+0000 7fce3af2b640 1 -- 172.21.15.37:0/3827650469 >> v1:172.21.15.117:6800/3703993921 conn(0x7fce1806e0b0 legacy=0x7fce18070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.282+0000 7fce3af2b640 10 monclient: shutdown 2024-03-25T18:09:37.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.282+0000 7fce3af2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:37.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.282+0000 7fce3af2b640 1 -- 172.21.15.37:0/3827650469 >> v1:172.21.15.37:6790/0 conn(0x7fce340b4d60 legacy=0x7fce341db2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:37.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.282+0000 7fce3af2b640 1 -- 172.21.15.37:0/3827650469 shutdown_connections 2024-03-25T18:09:37.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.282+0000 7fce3af2b640 1 -- 172.21.15.37:0/3827650469 >> 172.21.15.37:0/3827650469 conn(0x7fce340617a0 msgr2=0x7fce340b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:37.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.282+0000 7fce3af2b640 1 -- 172.21.15.37:0/3827650469 shutdown_connections 2024-03-25T18:09:37.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.282+0000 7fce3af2b640 1 -- 172.21.15.37:0/3827650469 wait complete. 2024-03-25T18:09:37.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:37.282+0000 7fce3af2b640 1 librados: shutdown 2024-03-25T18:09:37.294 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:44.101 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:44.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.215+0000 7f5820aa8640 10 monclient: get_monmap_and_config 2024-03-25T18:09:44.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.215+0000 7f5820aa8640 10 monclient: build_initial_monmap 2024-03-25T18:09:44.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.215+0000 7f5820aa8640 10 monclient: monmap: 2024-03-25T18:09:44.217 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:44.218 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:44.218 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:09:44.217554+0000 2024-03-25T18:09:44.218 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:09:44.217554+0000 2024-03-25T18:09:44.218 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:44.218 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:44.218 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:44.218 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:09:44.218 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:09:44.218 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.216+0000 7f5820aa8640 1 Processor -- start 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.216+0000 7f5820aa8640 1 -- start start 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.216+0000 7f5820aa8640 10 monclient: init 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.217+0000 7f5820aa8640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.217+0000 7f5820aa8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.217+0000 7f5820aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f581c0b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.217+0000 7f5820aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f581c0b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.217+0000 7f5820aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f581c14be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.217+0000 7f5820aa8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581c0af940 con 0x7f581c14be90 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.217+0000 7f5820aa8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581c0b0a70 con 0x7f581c0b1730 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.217+0000 7f5820aa8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581c0b0d50 con 0x7f581c0b4d60 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.217+0000 7f5820aa8640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:44.220 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.217+0000 7f5820aa8640 10 monclient(hunting): authenticate will time out at 6939.470709s 2024-03-25T18:09:44.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.224+0000 7f581ad9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f581c14be90 0x7f581c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34734/0 (socket says 172.21.15.37:34734) 2024-03-25T18:09:44.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.224+0000 7f581ad9d640 1 -- 172.21.15.37:0/3009031116 learned_addr learned my addr 172.21.15.37:0/3009031116 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:44.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.247+0000 7f5819d9b640 1 -- 172.21.15.37:0/3009031116 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3687893399 0 0) 0x7f581c0af940 con 0x7f581c14be90 2024-03-25T18:09:44.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.247+0000 7f5819d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:09:44.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.247+0000 7f5819d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:44.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.247+0000 7f5819d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:44.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.247+0000 7f5819d9b640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:09:44.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.247+0000 7f5819d9b640 1 -- 172.21.15.37:0/3009031116 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5808003520 con 0x7f581c14be90 2024-03-25T18:09:44.306 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:02:19.652 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:44.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 1 -- 172.21.15.37:0/3009031116 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 677215021 0 0) 0x7f5808003520 con 0x7f581c14be90 2024-03-25T18:09:44.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 1 -- 172.21.15.37:0/3009031116 >> v1:172.21.15.37:6790/0 conn(0x7f581c0b4d60 legacy=0x7f581c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:44.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 1 -- 172.21.15.37:0/3009031116 >> v1:172.21.15.117:6789/0 conn(0x7f581c0b1730 legacy=0x7f581c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:44.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 1 monclient: found mon.noname-a 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 1 -- 172.21.15.37:0/3009031116 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c0b1030 con 0x7f581c14be90 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 10 monclient: _finish_auth 0 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 1 -- 172.21.15.37:0/3009031116 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5814001a10 con 0x7f581c14be90 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5819d9b640 10 monclient: dump: 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:44.308 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:44.309 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:44.309 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:44.309 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:44.309 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:44.309 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:44.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5820aa8640 5 monclient: authenticate success, global_id 4305 2024-03-25T18:09:44.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.305+0000 7f5820aa8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:44.309 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:02:19.654 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:44.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.342+0000 7f5819d9b640 1 -- 172.21.15.37:0/3009031116 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5814001b70 con 0x7f581c14be90 2024-03-25T18:09:44.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.342+0000 7f5819d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:44.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.342+0000 7f5819d9b640 1 -- 172.21.15.37:0/3009031116 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5814001f80 con 0x7f581c14be90 2024-03-25T18:09:44.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.342+0000 7f5819d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.343+0000 7f5819d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.343+0000 7f5819d9b640 10 monclient: dump: 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.343+0000 7f5820aa8640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:44.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.343+0000 7f5820aa8640 4 set_mon_vals no callback set 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.343+0000 7f5820aa8640 10 monclient: shutdown 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.343+0000 7f5820aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.343+0000 7f5820aa8640 1 -- 172.21.15.37:0/3009031116 >> v1:172.21.15.37:6789/0 conn(0x7f581c14be90 legacy=0x7f581c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.343+0000 7f5820aa8640 1 -- 172.21.15.37:0/3009031116 shutdown_connections 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.343+0000 7f5820aa8640 1 -- 172.21.15.37:0/3009031116 >> 172.21.15.37:0/3009031116 conn(0x7f581c0617a0 msgr2=0x7f581c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.343+0000 7f5820aa8640 1 -- 172.21.15.37:0/3009031116 shutdown_connections 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.343+0000 7f5820aa8640 1 -- 172.21.15.37:0/3009031116 wait complete. 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.344+0000 7f5820aa8640 10 monclient: build_initial_monmap 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.344+0000 7f5820aa8640 10 monclient: monmap: 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:09:44.347 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 1 librados: starting msgr at 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 1 librados: starting objecter 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 1 Processor -- start 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 1 -- start start 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 1 librados: setting wanted keys 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 1 librados: calling monclient init 2024-03-25T18:09:44.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 10 monclient: init 2024-03-25T18:09:44.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:44.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:44.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 10 monclient(hunting): picked mon.noname-b con 0x7f581c0b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:44.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 10 monclient(hunting): picked mon.noname-c con 0x7f581c0b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:44.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 10 monclient(hunting): picked mon.noname-a con 0x7f581c14be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:44.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581c1307c0 con 0x7f581c14be90 2024-03-25T18:09:44.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.345+0000 7f5820aa8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581c138f90 con 0x7f581c0b1730 2024-03-25T18:09:44.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.346+0000 7f5820aa8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f581c132ab0 con 0x7f581c0b4d60 2024-03-25T18:09:44.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.346+0000 7f5820aa8640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:44.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.346+0000 7f5820aa8640 10 monclient(hunting): authenticate will time out at 6939.599295s 2024-03-25T18:09:44.355 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:02:19.700 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:44.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.353+0000 7f581ad9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f581c14be90 0x7f581c1300b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34738/0 (socket says 172.21.15.37:34738) 2024-03-25T18:09:44.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.353+0000 7f581ad9d640 1 -- 172.21.15.37:0/3984049942 learned_addr learned my addr 172.21.15.37:0/3984049942 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:44.363 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:02:19.709 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:44.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.365+0000 7f5812ffd640 10 client.?.objecter ms_handle_connect 0x7f581c14be90 2024-03-25T18:09:44.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.365+0000 7f5812ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:44.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.380+0000 7f5812ffd640 10 client.?.objecter ms_handle_connect 0x7f581c0b4d60 2024-03-25T18:09:44.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.380+0000 7f5812ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:44.382 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:02:19.728 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:44.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.399+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3473116762 0 0) 0x7f581c132ab0 con 0x7f581c0b4d60 2024-03-25T18:09:44.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.399+0000 7f5812ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:44.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.399+0000 7f5812ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:44.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.399+0000 7f5812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:44.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.399+0000 7f5812ffd640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:09:44.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.399+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57fc004eb0 con 0x7f581c0b4d60 2024-03-25T18:09:44.402 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:02:19.715 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:44.405 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:02:19.719 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:44.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.408+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891744178 0 0) 0x7f581c1307c0 con 0x7f581c14be90 2024-03-25T18:09:44.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.408+0000 7f5812ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:44.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.408+0000 7f5812ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:44.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.408+0000 7f5812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:44.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.408+0000 7f5812ffd640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:09:44.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.408+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f581c132ab0 con 0x7f581c14be90 2024-03-25T18:09:44.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.408+0000 7f5812ffd640 10 client.?.objecter ms_handle_connect 0x7f581c0b1730 2024-03-25T18:09:44.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.408+0000 7f5812ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:44.417 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:02:19.763 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.424+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 300193918 0 0) 0x7f57fc004eb0 con 0x7f581c0b4d60 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.424+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f581c1307c0 con 0x7f581c0b4d60 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.424+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5804001930 con 0x7f581c0b4d60 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.424+0000 7f5812ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.424+0000 7f5812ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.424+0000 7f5812ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:44.426 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:44.427 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:44.427 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:44.427 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:44.427 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:44.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.424+0000 7f5812ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:44.427 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:02:19.740 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:09:44.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.427+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365047673 0 0) 0x7f581c138f90 con 0x7f581c0b1730 2024-03-25T18:09:44.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.427+0000 7f5812ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:44.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.427+0000 7f5812ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:44.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.427+0000 7f5812ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:44.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.427+0000 7f5812ffd640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:09:44.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.427+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57fc004eb0 con 0x7f581c0b1730 2024-03-25T18:09:44.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.461+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 663647138 0 0) 0x7f581c132ab0 con 0x7f581c14be90 2024-03-25T18:09:44.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.461+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f581c138f90 con 0x7f581c14be90 2024-03-25T18:09:44.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.461+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5814001f40 con 0x7f581c14be90 2024-03-25T18:09:44.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.461+0000 7f5812ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:44.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.461+0000 7f5812ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:44.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.461+0000 7f5812ffd640 10 monclient(hunting): dump: 2024-03-25T18:09:44.463 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:44.463 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:44.463 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:44.464 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:44.464 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:44.464 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:44.464 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:44.464 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:44.464 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:44.464 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:44.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1141937805 0 0) 0x7f581c1307c0 con 0x7f581c0b4d60 2024-03-25T18:09:44.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 >> v1:172.21.15.117:6789/0 conn(0x7f581c0b1730 legacy=0x7f581c1393b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:44.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 >> v1:172.21.15.37:6789/0 conn(0x7f581c14be90 legacy=0x7f581c1300b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:44.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5812ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:44.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5812ffd640 1 monclient: found mon.c 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5812ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5812ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581c133a40 con 0x7f581c0b4d60 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5812ffd640 10 monclient: _finish_auth 0 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5812ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5812ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5820aa8640 5 monclient: authenticate success, global_id 4304 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5820aa8640 10 monclient: _renew_subs 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5820aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5820aa8640 1 -- 172.21.15.37:0/3984049942 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f581c1316c0 con 0x7f581c0b4d60 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5820aa8640 10 client.4304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5820aa8640 10 monclient: _renew_subs 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5820aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5820aa8640 1 -- 172.21.15.37:0/3984049942 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f581c137860 con 0x7f581c0b4d60 2024-03-25T18:09:44.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.465+0000 7f5820aa8640 1 librados: init done 2024-03-25T18:09:44.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.466+0000 7f5820aa8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:44.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.466+0000 7f5820aa8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:44.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.466+0000 7f5820aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:44.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.466+0000 7f5820aa8640 1 -- 172.21.15.37:0/3984049942 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f581c1319b0 con 0x7f581c0b4d60 2024-03-25T18:09:44.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.799+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58040035f0 con 0x7f581c0b4d60 2024-03-25T18:09:44.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.799+0000 7f5812ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:44.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.799+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5804003980 con 0x7f581c0b4d60 2024-03-25T18:09:44.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.799+0000 7f5812ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:44.801 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.799+0000 7f5812ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.799+0000 7f5812ffd640 10 monclient: dump: 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:44.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.799+0000 7f581959a640 4 set_mon_vals no callback set 2024-03-25T18:09:44.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.800+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f5804003ba0 con 0x7f581c0b4d60 2024-03-25T18:09:44.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.801+0000 7f581a59c640 10 monclient: get_auth_request con 0x7f57fc06dfb0 auth_method 0 2024-03-25T18:09:44.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.801+0000 7f5812ffd640 10 client.4304.objecter ms_handle_connect 0x7f57fc06dfb0 2024-03-25T18:09:44.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.810+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f58040535b0 con 0x7f581c0b4d60 2024-03-25T18:09:44.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.810+0000 7f5812ffd640 10 client.4304.objecter ms_dispatch 0x7f581c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:44.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.810+0000 7f5812ffd640 3 client.4304.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:44.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.810+0000 7f5812ffd640 3 client.4304.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:44.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.810+0000 7f5812ffd640 20 client.4304.objecter dump_active .. 0 homeless 2024-03-25T18:09:44.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.811+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f580404f0c0 con 0x7f581c0b4d60 2024-03-25T18:09:44.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.811+0000 7f5812ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:44.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:44.811+0000 7f5812ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:45.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.043+0000 7f5820aa8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:45.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.043+0000 7f5820aa8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:45.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.043+0000 7f5820aa8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:09:45.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.043+0000 7f5820aa8640 1 -- 172.21.15.37:0/3984049942 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f581c132c40 con 0x7f581c0b4d60 2024-03-25T18:09:45.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.064+0000 7f5812ffd640 1 -- 172.21.15.37:0/3984049942 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f5804050ff0 con 0x7f581c0b4d60 2024-03-25T18:09:45.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.064+0000 7f5812ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:45.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.064+0000 7f5812ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:45.066 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:09:45.066 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:45.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 10 librados: watch_flush enter 2024-03-25T18:09:45.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 10 librados: watch_flush exit 2024-03-25T18:09:45.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 20 client.4304.objecter shutdown clearing up homeless session... 2024-03-25T18:09:45.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 10 client.4304.objecter successfully canceled tick 2024-03-25T18:09:45.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 1 -- 172.21.15.37:0/3984049942 >> v1:172.21.15.117:6800/3703993921 conn(0x7f57fc06dfb0 legacy=0x7f57fc070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:45.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 10 monclient: shutdown 2024-03-25T18:09:45.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:45.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 1 -- 172.21.15.37:0/3984049942 >> v1:172.21.15.37:6790/0 conn(0x7f581c0b4d60 legacy=0x7f581c12f9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:45.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 1 -- 172.21.15.37:0/3984049942 shutdown_connections 2024-03-25T18:09:45.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 1 -- 172.21.15.37:0/3984049942 >> 172.21.15.37:0/3984049942 conn(0x7f581c0617a0 msgr2=0x7f581c13c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:45.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 1 -- 172.21.15.37:0/3984049942 shutdown_connections 2024-03-25T18:09:45.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 1 -- 172.21.15.37:0/3984049942 wait complete. 2024-03-25T18:09:45.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:45.067+0000 7f5820aa8640 1 librados: shutdown 2024-03-25T18:09:45.080 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:51.886 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:52.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.002+0000 7fe14e82e640 10 monclient: get_monmap_and_config 2024-03-25T18:09:52.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.002+0000 7fe14e82e640 10 monclient: build_initial_monmap 2024-03-25T18:09:52.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.002+0000 7fe14e82e640 10 monclient: monmap: 2024-03-25T18:09:52.004 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:52.004 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:52.004 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:09:52.003251+0000 2024-03-25T18:09:52.004 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:09:52.003251+0000 2024-03-25T18:09:52.004 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:52.004 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:52.005 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:52.005 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:09:52.005 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:09:52.005 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:52.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.003+0000 7fe14e82e640 1 Processor -- start 2024-03-25T18:09:52.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.003+0000 7fe14e82e640 1 -- start start 2024-03-25T18:09:52.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.003+0000 7fe14e82e640 10 monclient: init 2024-03-25T18:09:52.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.004+0000 7fe14e82e640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:52.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.004+0000 7fe14e82e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:52.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.004+0000 7fe14e82e640 10 monclient(hunting): picked mon.noname-b con 0x7fe1480a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:52.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.004+0000 7fe14e82e640 10 monclient(hunting): picked mon.noname-a con 0x7fe14813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:52.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.004+0000 7fe14e82e640 10 monclient(hunting): picked mon.noname-c con 0x7fe1481430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:52.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.004+0000 7fe14e82e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe148142520 con 0x7fe14813f710 2024-03-25T18:09:52.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.004+0000 7fe14e82e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1480b28a0 con 0x7fe1481430c0 2024-03-25T18:09:52.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.004+0000 7fe14e82e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1480b2a30 con 0x7fe1480a98f0 2024-03-25T18:09:52.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.004+0000 7fe14e82e640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:52.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.004+0000 7fe14e82e640 10 monclient(hunting): authenticate will time out at 6947.256473s 2024-03-25T18:09:52.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.008+0000 7fe14cdcb640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe1481430c0 0x7fe14814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:44820/0 (socket says 172.21.15.37:44820) 2024-03-25T18:09:52.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.008+0000 7fe14cdcb640 1 -- 172.21.15.37:0/334498717 learned_addr learned my addr 172.21.15.37:0/334498717 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:52.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.026+0000 7fe146ffd640 1 -- 172.21.15.37:0/334498717 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3566743254 0 0) 0x7fe1480b28a0 con 0x7fe1481430c0 2024-03-25T18:09:52.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.026+0000 7fe146ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:09:52.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.026+0000 7fe146ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:52.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.026+0000 7fe146ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:52.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.026+0000 7fe146ffd640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:09:52.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.026+0000 7fe146ffd640 1 -- 172.21.15.37:0/334498717 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe130003520 con 0x7fe1481430c0 2024-03-25T18:09:52.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 1 -- 172.21.15.37:0/334498717 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3010832658 0 0) 0x7fe130003520 con 0x7fe1481430c0 2024-03-25T18:09:52.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 1 -- 172.21.15.37:0/334498717 >> v1:172.21.15.37:6790/0 conn(0x7fe1480a98f0 legacy=0x7fe1480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:52.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 1 -- 172.21.15.37:0/334498717 >> v1:172.21.15.37:6789/0 conn(0x7fe14813f710 legacy=0x7fe1481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:52.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:09:52.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 1 monclient: found mon.noname-c 2024-03-25T18:09:52.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:52.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:09:52.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 1 -- 172.21.15.37:0/334498717 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1480b44c0 con 0x7fe1481430c0 2024-03-25T18:09:52.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 10 monclient: _finish_auth 0 2024-03-25T18:09:52.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:09:52.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:52.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 1 -- 172.21.15.37:0/334498717 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe140001a10 con 0x7fe1481430c0 2024-03-25T18:09:52.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:52.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:09:52.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe146ffd640 10 monclient: dump: 2024-03-25T18:09:52.037 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:52.037 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:52.037 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:52.037 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:52.038 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:52.038 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:52.038 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:52.038 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:52.038 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:52.038 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:52.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe14e82e640 5 monclient: authenticate success, global_id 4312 2024-03-25T18:09:52.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.033+0000 7fe14e82e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe146ffd640 1 -- 172.21.15.37:0/334498717 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe140001b70 con 0x7fe1481430c0 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe146ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe146ffd640 1 -- 172.21.15.37:0/334498717 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe140001f80 con 0x7fe1481430c0 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe146ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe146ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe146ffd640 10 monclient: dump: 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:52.043 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:52.044 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:52.044 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:52.044 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:52.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe14e82e640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:52.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe14e82e640 4 set_mon_vals no callback set 2024-03-25T18:09:52.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe14e82e640 10 monclient: shutdown 2024-03-25T18:09:52.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe14e82e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe14e82e640 1 -- 172.21.15.37:0/334498717 >> v1:172.21.15.117:6789/0 conn(0x7fe1481430c0 legacy=0x7fe14814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe14e82e640 1 -- 172.21.15.37:0/334498717 shutdown_connections 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.042+0000 7fe14e82e640 1 -- 172.21.15.37:0/334498717 >> 172.21.15.37:0/334498717 conn(0x7fe1480b7ea0 msgr2=0x7fe1480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.043+0000 7fe14e82e640 1 -- 172.21.15.37:0/334498717 shutdown_connections 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.043+0000 7fe14e82e640 1 -- 172.21.15.37:0/334498717 wait complete. 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.043+0000 7fe14e82e640 10 monclient: build_initial_monmap 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.043+0000 7fe14e82e640 10 monclient: monmap: 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:52.047 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.044+0000 7fe14e82e640 1 librados: starting msgr at 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.044+0000 7fe14e82e640 1 librados: starting objecter 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.044+0000 7fe14e82e640 1 Processor -- start 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.044+0000 7fe14e82e640 1 -- start start 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.044+0000 7fe14e82e640 1 librados: setting wanted keys 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.044+0000 7fe14e82e640 1 librados: calling monclient init 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.044+0000 7fe14e82e640 10 monclient: init 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.045+0000 7fe14e82e640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.045+0000 7fe14e82e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.045+0000 7fe14e82e640 10 monclient(hunting): picked mon.noname-b con 0x7fe1480a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.045+0000 7fe14e82e640 10 monclient(hunting): picked mon.noname-c con 0x7fe14813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:52.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.045+0000 7fe14e82e640 10 monclient(hunting): picked mon.noname-a con 0x7fe1481430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:52.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.045+0000 7fe14e82e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe148138eb0 con 0x7fe1481430c0 2024-03-25T18:09:52.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.045+0000 7fe14e82e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1481376a0 con 0x7fe1480a98f0 2024-03-25T18:09:52.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.045+0000 7fe14e82e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1481d7c10 con 0x7fe14813f710 2024-03-25T18:09:52.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.045+0000 7fe14e82e640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:52.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.045+0000 7fe14e82e640 10 monclient(hunting): authenticate will time out at 6947.298271s 2024-03-25T18:09:52.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.056+0000 7fe14cdcb640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe1481430c0 0x7fe148132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48432/0 (socket says 172.21.15.37:48432) 2024-03-25T18:09:52.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.056+0000 7fe14cdcb640 1 -- 172.21.15.37:0/1166884597 learned_addr learned my addr 172.21.15.37:0/1166884597 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:52.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.060+0000 7fe144ff9640 10 client.?.objecter ms_handle_connect 0x7fe1481430c0 2024-03-25T18:09:52.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.060+0000 7fe144ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:52.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.060+0000 7fe144ff9640 10 client.?.objecter ms_handle_connect 0x7fe14813f710 2024-03-25T18:09:52.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.060+0000 7fe144ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:52.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.065+0000 7fe144ff9640 10 client.?.objecter ms_handle_connect 0x7fe1480a98f0 2024-03-25T18:09:52.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.065+0000 7fe144ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:52.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.071+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860163287 0 0) 0x7fe1481376a0 con 0x7fe1480a98f0 2024-03-25T18:09:52.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.071+0000 7fe144ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:52.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.071+0000 7fe144ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:52.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.071+0000 7fe144ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:52.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.071+0000 7fe144ff9640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:09:52.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.071+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe134004eb0 con 0x7fe1480a98f0 2024-03-25T18:09:52.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.076+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449492762 0 0) 0x7fe148138eb0 con 0x7fe1481430c0 2024-03-25T18:09:52.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.076+0000 7fe144ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:52.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.076+0000 7fe144ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:52.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.076+0000 7fe144ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:52.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.076+0000 7fe144ff9640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:09:52.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.076+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1481376a0 con 0x7fe1481430c0 2024-03-25T18:09:52.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.081+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1871974249 0 0) 0x7fe1481d7c10 con 0x7fe14813f710 2024-03-25T18:09:52.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.081+0000 7fe144ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:09:52.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.081+0000 7fe144ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:52.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.081+0000 7fe144ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:52.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.081+0000 7fe144ff9640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:09:52.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.081+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe148138eb0 con 0x7fe14813f710 2024-03-25T18:09:52.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.085+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 447390705 0 0) 0x7fe134004eb0 con 0x7fe1480a98f0 2024-03-25T18:09:52.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.085+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe1481d7c10 con 0x7fe1480a98f0 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.086+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe138001930 con 0x7fe1480a98f0 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.086+0000 7fe144ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.086+0000 7fe144ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.086+0000 7fe144ff9640 10 monclient(hunting): dump: 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:52.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.086+0000 7fe144ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:52.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.091+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1275015641 0 0) 0x7fe1481376a0 con 0x7fe1481430c0 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.091+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe134004eb0 con 0x7fe1481430c0 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.091+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe140001ec0 con 0x7fe1481430c0 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.091+0000 7fe144ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.091+0000 7fe144ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.091+0000 7fe144ff9640 10 monclient(hunting): dump: 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:52.092 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:52.093 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:52.093 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:52.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1378201699 0 0) 0x7fe134004eb0 con 0x7fe1481430c0 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 >> v1:172.21.15.37:6790/0 conn(0x7fe14813f710 legacy=0x7fe148136a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 >> v1:172.21.15.117:6789/0 conn(0x7fe1480a98f0 legacy=0x7fe1481d7500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe144ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe144ff9640 1 monclient: found mon.a 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe144ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe144ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe148137140 con 0x7fe1481430c0 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe144ff9640 10 monclient: _finish_auth 0 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe144ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe144ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe14e82e640 5 monclient: authenticate success, global_id 4314 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe14e82e640 10 monclient: _renew_subs 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe14e82e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:09:52.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.103+0000 7fe14e82e640 1 -- 172.21.15.37:0/1166884597 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe148137bc0 con 0x7fe1481430c0 2024-03-25T18:09:52.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.104+0000 7fe14e82e640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:52.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.104+0000 7fe14e82e640 10 monclient: _renew_subs 2024-03-25T18:09:52.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.104+0000 7fe14e82e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:09:52.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.104+0000 7fe14e82e640 1 -- 172.21.15.37:0/1166884597 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe14813bb00 con 0x7fe1481430c0 2024-03-25T18:09:52.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.104+0000 7fe14e82e640 1 librados: init done 2024-03-25T18:09:52.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.104+0000 7fe14e82e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:52.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.104+0000 7fe14e82e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:52.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.104+0000 7fe14e82e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:09:52.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.104+0000 7fe14e82e640 1 -- 172.21.15.37:0/1166884597 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe148137850 con 0x7fe1481430c0 2024-03-25T18:09:52.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.428+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe1400020e0 con 0x7fe1481430c0 2024-03-25T18:09:52.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.428+0000 7fe144ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:52.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.428+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe140004e10 con 0x7fe1481430c0 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.428+0000 7fe144ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.428+0000 7fe144ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.428+0000 7fe144ff9640 10 monclient: dump: 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:52.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.428+0000 7fe1467fc640 4 set_mon_vals no callback set 2024-03-25T18:09:52.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.429+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fe14001ade0 con 0x7fe1481430c0 2024-03-25T18:09:52.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.430+0000 7fe147fff640 10 monclient: get_auth_request con 0x7fe13406df70 auth_method 0 2024-03-25T18:09:52.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.430+0000 7fe144ff9640 10 client.4314.objecter ms_handle_connect 0x7fe13406df70 2024-03-25T18:09:52.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.436+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fe140054a50 con 0x7fe1481430c0 2024-03-25T18:09:52.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.436+0000 7fe144ff9640 10 client.4314.objecter ms_dispatch 0x7fe148000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:09:52.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.436+0000 7fe144ff9640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:09:52.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.436+0000 7fe144ff9640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:09:52.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.436+0000 7fe144ff9640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-25T18:09:52.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.436+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe140050390 con 0x7fe1481430c0 2024-03-25T18:09:52.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.436+0000 7fe144ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:52.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.436+0000 7fe144ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:09:52.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.673+0000 7fe14e82e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:52.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.673+0000 7fe14e82e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:52.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.673+0000 7fe14e82e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:09:52.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.673+0000 7fe14e82e640 1 -- 172.21.15.37:0/1166884597 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe1480627b0 con 0x7fe1481430c0 2024-03-25T18:09:52.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.713+0000 7fe144ff9640 1 -- 172.21.15.37:0/1166884597 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fe140052490 con 0x7fe1481430c0 2024-03-25T18:09:52.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.713+0000 7fe144ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:09:52.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.713+0000 7fe144ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:09:52.714 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:09:52.714 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:09:52.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 10 librados: watch_flush enter 2024-03-25T18:09:52.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 10 librados: watch_flush exit 2024-03-25T18:09:52.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-25T18:09:52.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 10 client.4314.objecter successfully canceled tick 2024-03-25T18:09:52.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 1 -- 172.21.15.37:0/1166884597 >> v1:172.21.15.117:6800/3703993921 conn(0x7fe13406df70 legacy=0x7fe134070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:52.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 10 monclient: shutdown 2024-03-25T18:09:52.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:52.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 1 -- 172.21.15.37:0/1166884597 >> v1:172.21.15.37:6789/0 conn(0x7fe1481430c0 legacy=0x7fe148132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:52.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 1 -- 172.21.15.37:0/1166884597 shutdown_connections 2024-03-25T18:09:52.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 1 -- 172.21.15.37:0/1166884597 >> 172.21.15.37:0/1166884597 conn(0x7fe1480b7ea0 msgr2=0x7fe148074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:52.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 1 -- 172.21.15.37:0/1166884597 shutdown_connections 2024-03-25T18:09:52.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 1 -- 172.21.15.37:0/1166884597 wait complete. 2024-03-25T18:09:52.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:52.715+0000 7fe14e82e640 1 librados: shutdown 2024-03-25T18:09:52.727 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:09:59.533 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:09:59.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.650+0000 7f73f1efb640 10 monclient: get_monmap_and_config 2024-03-25T18:09:59.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.650+0000 7f73f1efb640 10 monclient: build_initial_monmap 2024-03-25T18:09:59.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.650+0000 7f73f1efb640 10 monclient: monmap: 2024-03-25T18:09:59.652 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:59.652 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:59.652 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:09:59.651603+0000 2024-03-25T18:09:59.652 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:09:59.651603+0000 2024-03-25T18:09:59.652 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:59.652 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:59.652 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:59.652 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:09:59.652 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:09:59.652 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:59.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.651+0000 7f73f1efb640 1 Processor -- start 2024-03-25T18:09:59.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.651+0000 7f73f1efb640 1 -- start start 2024-03-25T18:09:59.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.651+0000 7f73f1efb640 10 monclient: init 2024-03-25T18:09:59.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.652+0000 7f73f1efb640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:59.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.652+0000 7f73f1efb640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:09:59.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.652+0000 7f73f1efb640 10 monclient(hunting): picked mon.noname-b con 0x7f73ec0b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:59.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.652+0000 7f73f1efb640 10 monclient(hunting): picked mon.noname-a con 0x7f73ec0b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:59.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.652+0000 7f73f1efb640 10 monclient(hunting): picked mon.noname-c con 0x7f73ec14be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:59.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.652+0000 7f73f1efb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec0af940 con 0x7f73ec0b1730 2024-03-25T18:09:59.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.652+0000 7f73f1efb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec0b0a70 con 0x7f73ec14be90 2024-03-25T18:09:59.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.652+0000 7f73f1efb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec0b0d50 con 0x7f73ec0b4d60 2024-03-25T18:09:59.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.652+0000 7f73f1efb640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:59.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.652+0000 7f73f1efb640 10 monclient(hunting): authenticate will time out at 6954.904971s 2024-03-25T18:09:59.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.659+0000 7f73eb7fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f73ec0b4d60 0x7f73ec13f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:60372/0 (socket says 172.21.15.37:60372) 2024-03-25T18:09:59.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.659+0000 7f73eb7fe640 1 -- 172.21.15.37:0/4104255237 learned_addr learned my addr 172.21.15.37:0/4104255237 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:59.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 1 -- 172.21.15.37:0/4104255237 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 824811785 0 0) 0x7f73ec0b0a70 con 0x7f73ec14be90 2024-03-25T18:09:59.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:59.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:59.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:59.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:09:59.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 1 -- 172.21.15.37:0/4104255237 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d4003520 con 0x7f73ec14be90 2024-03-25T18:09:59.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 1 -- 172.21.15.37:0/4104255237 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73284882 0 0) 0x7f73ec0b0d50 con 0x7f73ec0b4d60 2024-03-25T18:09:59.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:09:59.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:59.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:59.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:09:59.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.676+0000 7f73ea7fc640 1 -- 172.21.15.37:0/4104255237 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73ec0b0a70 con 0x7f73ec0b4d60 2024-03-25T18:09:59.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 1 -- 172.21.15.37:0/4104255237 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1899685531 0 0) 0x7f73ec0b0a70 con 0x7f73ec0b4d60 2024-03-25T18:09:59.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 1 -- 172.21.15.37:0/4104255237 >> v1:172.21.15.117:6789/0 conn(0x7f73ec14be90 legacy=0x7f73ec14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:59.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 1 -- 172.21.15.37:0/4104255237 >> v1:172.21.15.37:6789/0 conn(0x7f73ec0b1730 legacy=0x7f73ec0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 1 monclient: found mon.noname-b 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 1 -- 172.21.15.37:0/4104255237 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec0b1030 con 0x7f73ec0b4d60 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 10 monclient: _finish_auth 0 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 1 -- 172.21.15.37:0/4104255237 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f73dc001670 con 0x7f73ec0b4d60 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:59.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73ea7fc640 10 monclient: dump: 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73f1efb640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:09:59.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.691+0000 7f73f1efb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:09:59.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.710+0000 7f73ea7fc640 1 -- 172.21.15.37:0/4104255237 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73dc0017d0 con 0x7f73ec0b4d60 2024-03-25T18:09:59.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.710+0000 7f73ea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:09:59.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.710+0000 7f73ea7fc640 1 -- 172.21.15.37:0/4104255237 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f73dc001ba0 con 0x7f73ec0b4d60 2024-03-25T18:09:59.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.710+0000 7f73ea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:59.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.710+0000 7f73ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:09:59.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.710+0000 7f73ea7fc640 10 monclient: dump: 2024-03-25T18:09:59.712 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:59.712 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:59.712 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:59.712 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:59.712 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:59.712 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:59.712 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:59.712 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:59.713 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:59.713 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:59.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.710+0000 7f73f1efb640 10 monclient: get_monmap_and_config success 2024-03-25T18:09:59.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.710+0000 7f73f1efb640 4 set_mon_vals no callback set 2024-03-25T18:09:59.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.711+0000 7f73f1efb640 10 monclient: shutdown 2024-03-25T18:09:59.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.711+0000 7f73f1efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:09:59.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.711+0000 7f73f1efb640 1 -- 172.21.15.37:0/4104255237 >> v1:172.21.15.37:6790/0 conn(0x7f73ec0b4d60 legacy=0x7f73ec13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:59.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.711+0000 7f73f1efb640 1 -- 172.21.15.37:0/4104255237 shutdown_connections 2024-03-25T18:09:59.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.711+0000 7f73f1efb640 1 -- 172.21.15.37:0/4104255237 >> 172.21.15.37:0/4104255237 conn(0x7f73ec0617a0 msgr2=0x7f73ec061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.711+0000 7f73f1efb640 1 -- 172.21.15.37:0/4104255237 shutdown_connections 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.711+0000 7f73f1efb640 1 -- 172.21.15.37:0/4104255237 wait complete. 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.712+0000 7f73f1efb640 10 monclient: build_initial_monmap 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.712+0000 7f73f1efb640 10 monclient: monmap: 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:09:59.716 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.712+0000 7f73f1efb640 1 librados: starting msgr at 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.712+0000 7f73f1efb640 1 librados: starting objecter 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 1 Processor -- start 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 1 -- start start 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 1 librados: setting wanted keys 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 1 librados: calling monclient init 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 10 monclient: init 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 10 monclient: _reopen_session rank -1 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 10 monclient(hunting): picked mon.noname-b con 0x7f73ec0b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 10 monclient(hunting): picked mon.noname-a con 0x7f73ec0b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 10 monclient(hunting): picked mon.noname-c con 0x7f73ec14be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec141d30 con 0x7f73ec0b4d60 2024-03-25T18:09:59.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec140a30 con 0x7f73ec0b1730 2024-03-25T18:09:59.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73ec140ec0 con 0x7f73ec14be90 2024-03-25T18:09:59.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 10 monclient(hunting): _renew_subs 2024-03-25T18:09:59.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.713+0000 7f73f1efb640 10 monclient(hunting): authenticate will time out at 6954.966260s 2024-03-25T18:09:59.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.721+0000 7f73eb7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f73ec0b1730 0x7f73ec131da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:52554/0 (socket says 172.21.15.37:52554) 2024-03-25T18:09:59.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.721+0000 7f73eb7fe640 1 -- 172.21.15.37:0/1560451940 learned_addr learned my addr 172.21.15.37:0/1560451940 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:09:59.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.721+0000 7f73d3fff640 10 client.?.objecter ms_handle_connect 0x7f73ec0b1730 2024-03-25T18:09:59.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.721+0000 7f73d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:59.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.723+0000 7f73d3fff640 10 client.?.objecter ms_handle_connect 0x7f73ec0b4d60 2024-03-25T18:09:59.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.723+0000 7f73d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:59.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.726+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595427413 0 0) 0x7f73ec140a30 con 0x7f73ec0b1730 2024-03-25T18:09:59.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.727+0000 7f73d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:59.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.727+0000 7f73d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:59.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.727+0000 7f73d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:59.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.727+0000 7f73d3fff640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:09:59.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.727+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d8004eb0 con 0x7f73ec0b1730 2024-03-25T18:09:59.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.729+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473504802 0 0) 0x7f73ec141d30 con 0x7f73ec0b4d60 2024-03-25T18:09:59.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.729+0000 7f73d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:59.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.729+0000 7f73d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:59.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.729+0000 7f73d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:59.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.729+0000 7f73d3fff640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:09:59.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.729+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73ec140a30 con 0x7f73ec0b4d60 2024-03-25T18:09:59.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.729+0000 7f73d3fff640 10 client.?.objecter ms_handle_connect 0x7f73ec14be90 2024-03-25T18:09:59.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.729+0000 7f73d3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.734+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2900413881 0 0) 0x7f73d8004eb0 con 0x7f73ec0b1730 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.734+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73ec141d30 con 0x7f73ec0b1730 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.734+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f73dc001ba0 con 0x7f73ec0b1730 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.734+0000 7f73d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.734+0000 7f73d3fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.734+0000 7f73d3fff640 10 monclient(hunting): dump: 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:59.735 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:59.736 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:59.736 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:59.736 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:59.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.734+0000 7f73d3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:09:59.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.741+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2077051519 0 0) 0x7f73ec140a30 con 0x7f73ec0b4d60 2024-03-25T18:09:59.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.741+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73d8004eb0 con 0x7f73ec0b4d60 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.741+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f73e00018a0 con 0x7f73ec0b4d60 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.741+0000 7f73d3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.741+0000 7f73d3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.741+0000 7f73d3fff640 10 monclient(hunting): dump: 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:09:59.743 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:09:59.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.743+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142610830 0 0) 0x7f73ec140ec0 con 0x7f73ec14be90 2024-03-25T18:09:59.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.743+0000 7f73d3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:09:59.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.743+0000 7f73d3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:09:59.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.743+0000 7f73d3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:09:59.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.743+0000 7f73d3fff640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:09:59.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.743+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73ec140a30 con 0x7f73ec14be90 2024-03-25T18:09:59.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 429357094 0 0) 0x7f73ec141d30 con 0x7f73ec0b1730 2024-03-25T18:09:59.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 >> v1:172.21.15.37:6790/0 conn(0x7f73ec14be90 legacy=0x7f73ec1e2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:59.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 >> v1:172.21.15.37:6789/0 conn(0x7f73ec0b4d60 legacy=0x7f73ec1dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73d3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73d3fff640 1 monclient: found mon.b 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73d3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73d3fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73ec137f40 con 0x7f73ec0b1730 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73d3fff640 10 monclient: _finish_auth 0 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73d3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73d3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73f1efb640 5 monclient: authenticate success, global_id 4321 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73f1efb640 10 monclient: _renew_subs 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73f1efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73f1efb640 1 -- 172.21.15.37:0/1560451940 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73ec1381f0 con 0x7f73ec0b1730 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73f1efb640 10 client.4321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73f1efb640 10 monclient: _renew_subs 2024-03-25T18:09:59.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73f1efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:09:59.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73f1efb640 1 -- 172.21.15.37:0/1560451940 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73ec1326e0 con 0x7f73ec0b1730 2024-03-25T18:09:59.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.753+0000 7f73f1efb640 1 librados: init done 2024-03-25T18:09:59.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.754+0000 7f73f1efb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:59.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.754+0000 7f73f1efb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:09:59.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.754+0000 7f73f1efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:09:59.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:09:59.754+0000 7f73f1efb640 1 -- 172.21.15.37:0/1560451940 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73ec1421c0 con 0x7f73ec0b1730 2024-03-25T18:10:00.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.033+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73dc004a90 con 0x7f73ec0b1730 2024-03-25T18:10:00.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.033+0000 7f73d3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:00.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.033+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f73dc004d00 con 0x7f73ec0b1730 2024-03-25T18:10:00.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.033+0000 7f73d3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:00.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.033+0000 7f73d3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:00.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.033+0000 7f73d3fff640 10 monclient: dump: 2024-03-25T18:10:00.035 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:00.035 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:00.036 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:00.036 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:00.036 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:00.036 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:00.036 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:00.036 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:00.036 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:00.036 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:00.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.033+0000 7f73e9ffb640 4 set_mon_vals no callback set 2024-03-25T18:10:00.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.034+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f73dc004f20 con 0x7f73ec0b1730 2024-03-25T18:10:00.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.035+0000 7f73eaffd640 10 monclient: get_auth_request con 0x7f73d806df50 auth_method 0 2024-03-25T18:10:00.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.035+0000 7f73d3fff640 10 client.4321.objecter ms_handle_connect 0x7f73d806df50 2024-03-25T18:10:00.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.040+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f73dc054820 con 0x7f73ec0b1730 2024-03-25T18:10:00.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.040+0000 7f73d3fff640 10 client.4321.objecter ms_dispatch 0x7f73ec000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:00.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.040+0000 7f73d3fff640 3 client.4321.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:00.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.040+0000 7f73d3fff640 3 client.4321.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:00.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.040+0000 7f73d3fff640 20 client.4321.objecter dump_active .. 0 homeless 2024-03-25T18:10:00.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.040+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f73dc050330 con 0x7f73ec0b1730 2024-03-25T18:10:00.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.040+0000 7f73d3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:00.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.040+0000 7f73d3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:00.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.277+0000 7f73f1efb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:00.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.277+0000 7f73f1efb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:00.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.277+0000 7f73f1efb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:00.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.277+0000 7f73f1efb640 1 -- 172.21.15.37:0/1560451940 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73ec0627b0 con 0x7f73ec0b1730 2024-03-25T18:10:00.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.307+0000 7f73d3fff640 1 -- 172.21.15.37:0/1560451940 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f73dc052260 con 0x7f73ec0b1730 2024-03-25T18:10:00.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.307+0000 7f73d3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:00.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.307+0000 7f73d3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:00.308 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:10:00.308 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.309+0000 7f73f1efb640 10 librados: watch_flush enter 2024-03-25T18:10:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.309+0000 7f73f1efb640 10 librados: watch_flush exit 2024-03-25T18:10:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.309+0000 7f73f1efb640 20 client.4321.objecter shutdown clearing up homeless session... 2024-03-25T18:10:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.309+0000 7f73f1efb640 10 client.4321.objecter successfully canceled tick 2024-03-25T18:10:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.310+0000 7f73f1efb640 1 -- 172.21.15.37:0/1560451940 >> v1:172.21.15.117:6800/3703993921 conn(0x7f73d806df50 legacy=0x7f73d8070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.310+0000 7f73f1efb640 10 monclient: shutdown 2024-03-25T18:10:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.310+0000 7f73f1efb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.310+0000 7f73f1efb640 1 -- 172.21.15.37:0/1560451940 >> v1:172.21.15.117:6789/0 conn(0x7f73ec0b1730 legacy=0x7f73ec131da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:00.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.310+0000 7f73f1efb640 1 -- 172.21.15.37:0/1560451940 shutdown_connections 2024-03-25T18:10:00.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.310+0000 7f73f1efb640 1 -- 172.21.15.37:0/1560451940 >> 172.21.15.37:0/1560451940 conn(0x7f73ec0617a0 msgr2=0x7f73ec0b63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:00.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.310+0000 7f73f1efb640 1 -- 172.21.15.37:0/1560451940 shutdown_connections 2024-03-25T18:10:00.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.310+0000 7f73f1efb640 1 -- 172.21.15.37:0/1560451940 wait complete. 2024-03-25T18:10:00.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:00.310+0000 7f73f1efb640 1 librados: shutdown 2024-03-25T18:10:00.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:04.382 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:02:39.727 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:04.388 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:02:39.734 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:04.445 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:02:39.790 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:04.450 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:02:39.795 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:04.459 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:02:39.805 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:04.479 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:02:39.792 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:04.485 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:02:39.798 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:04.489 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:02:39.834 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:04.505 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:02:39.818 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:07.129 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:07.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.244+0000 7f8a33625640 10 monclient: get_monmap_and_config 2024-03-25T18:10:07.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.244+0000 7f8a33625640 10 monclient: build_initial_monmap 2024-03-25T18:10:07.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.244+0000 7f8a33625640 10 monclient: monmap: 2024-03-25T18:10:07.245 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:07.245 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:07.245 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:10:07.245148+0000 2024-03-25T18:10:07.245 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:10:07.245148+0000 2024-03-25T18:10:07.245 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:07.245 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:07.245 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:07.245 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:10:07.246 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:10:07.246 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:07.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.244+0000 7f8a33625640 1 Processor -- start 2024-03-25T18:10:07.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.245+0000 7f8a33625640 1 -- start start 2024-03-25T18:10:07.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.245+0000 7f8a33625640 10 monclient: init 2024-03-25T18:10:07.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.245+0000 7f8a33625640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:07.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.245+0000 7f8a33625640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:07.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.245+0000 7f8a33625640 10 monclient(hunting): picked mon.noname-c con 0x7f8a2c0ae6e0 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:07.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.245+0000 7f8a33625640 10 monclient(hunting): picked mon.noname-a con 0x7f8a2c06f080 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:07.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.245+0000 7f8a33625640 10 monclient(hunting): picked mon.noname-b con 0x7f8a2c0704e0 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:07.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.245+0000 7f8a33625640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c0718d0 con 0x7f8a2c06f080 2024-03-25T18:10:07.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.246+0000 7f8a33625640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c06fa30 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.246+0000 7f8a33625640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c070090 con 0x7f8a2c0704e0 2024-03-25T18:10:07.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.246+0000 7f8a33625640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:07.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.246+0000 7f8a33625640 10 monclient(hunting): authenticate will time out at 6962.498310s 2024-03-25T18:10:07.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.259+0000 7f8a313c1640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f8a2c0ae6e0 0x7f8a2c13c320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:57028/0 (socket says 172.21.15.37:57028) 2024-03-25T18:10:07.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.259+0000 7f8a313c1640 1 -- 172.21.15.37:0/2837309630 learned_addr learned my addr 172.21.15.37:0/2837309630 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:07.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.268+0000 7f8a23fff640 1 -- 172.21.15.37:0/2837309630 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2762348613 0 0) 0x7f8a2c06fa30 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.269+0000 7f8a23fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:07.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.269+0000 7f8a23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:07.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.269+0000 7f8a23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:07.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.269+0000 7f8a23fff640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:10:07.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.269+0000 7f8a23fff640 1 -- 172.21.15.37:0/2837309630 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a18003520 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.277+0000 7f8a23fff640 1 -- 172.21.15.37:0/2837309630 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3285083162 0 0) 0x7f8a2c070090 con 0x7f8a2c0704e0 2024-03-25T18:10:07.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.277+0000 7f8a23fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:07.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.277+0000 7f8a23fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:07.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.277+0000 7f8a23fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:07.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.277+0000 7f8a23fff640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:10:07.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.277+0000 7f8a23fff640 1 -- 172.21.15.37:0/2837309630 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a2c06fa30 con 0x7f8a2c0704e0 2024-03-25T18:10:07.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 1 -- 172.21.15.37:0/2837309630 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 293226138 0 0) 0x7f8a18003520 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 1 -- 172.21.15.37:0/2837309630 >> v1:172.21.15.37:6790/0 conn(0x7f8a2c0704e0 legacy=0x7f8a2c14a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 1 -- 172.21.15.37:0/2837309630 >> v1:172.21.15.37:6789/0 conn(0x7f8a2c06f080 legacy=0x7f8a2c071c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 1 monclient: found mon.noname-c 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 1 -- 172.21.15.37:0/2837309630 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c06f510 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 10 monclient: _finish_auth 0 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 1 -- 172.21.15.37:0/2837309630 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8a28001690 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:07.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a23fff640 10 monclient: dump: 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a33625640 5 monclient: authenticate success, global_id 4324 2024-03-25T18:10:07.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.278+0000 7f8a33625640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:07.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.307+0000 7f8a23fff640 1 -- 172.21.15.37:0/2837309630 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a280017f0 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.307+0000 7f8a23fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:07.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.307+0000 7f8a23fff640 1 -- 172.21.15.37:0/2837309630 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8a28001c00 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.307+0000 7f8a33625640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:07.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.307+0000 7f8a33625640 4 set_mon_vals no callback set 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.307+0000 7f8a23fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.307+0000 7f8a23fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.307+0000 7f8a23fff640 10 monclient: dump: 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:07.309 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:07.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.308+0000 7f8a33625640 10 monclient: shutdown 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.308+0000 7f8a33625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.308+0000 7f8a33625640 1 -- 172.21.15.37:0/2837309630 >> v1:172.21.15.117:6789/0 conn(0x7f8a2c0ae6e0 legacy=0x7f8a2c13c320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.308+0000 7f8a33625640 1 -- 172.21.15.37:0/2837309630 shutdown_connections 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.308+0000 7f8a33625640 1 -- 172.21.15.37:0/2837309630 >> 172.21.15.37:0/2837309630 conn(0x7f8a2c06b630 msgr2=0x7f8a2c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.308+0000 7f8a33625640 1 -- 172.21.15.37:0/2837309630 shutdown_connections 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.308+0000 7f8a33625640 1 -- 172.21.15.37:0/2837309630 wait complete. 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.309+0000 7f8a33625640 10 monclient: build_initial_monmap 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.309+0000 7f8a33625640 10 monclient: monmap: 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:07.312 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.309+0000 7f8a33625640 1 librados: starting msgr at 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.309+0000 7f8a33625640 1 librados: starting objecter 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 1 Processor -- start 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 1 -- start start 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 1 librados: setting wanted keys 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 1 librados: calling monclient init 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 10 monclient: init 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:07.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 10 monclient(hunting): picked mon.noname-a con 0x7f8a2c0704e0 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:07.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 10 monclient(hunting): picked mon.noname-c con 0x7f8a2c0ae6e0 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:07.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 10 monclient(hunting): picked mon.noname-b con 0x7f8a2c1d5a30 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:07.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c0abb60 con 0x7f8a2c0704e0 2024-03-25T18:10:07.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c1d2e60 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8a2c0ac080 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:07.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.310+0000 7f8a33625640 10 monclient(hunting): authenticate will time out at 6962.563048s 2024-03-25T18:10:07.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.316+0000 7f8a313c1640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8a2c0704e0 0x7f8a2c1d5320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43682/0 (socket says 172.21.15.37:43682) 2024-03-25T18:10:07.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.316+0000 7f8a313c1640 1 -- 172.21.15.37:0/3933249389 learned_addr learned my addr 172.21.15.37:0/3933249389 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:07.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.319+0000 7f8a21ffb640 10 client.?.objecter ms_handle_connect 0x7f8a2c0704e0 2024-03-25T18:10:07.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.319+0000 7f8a21ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:07.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.323+0000 7f8a21ffb640 10 client.?.objecter ms_handle_connect 0x7f8a2c1d5a30 2024-03-25T18:10:07.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.323+0000 7f8a21ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:07.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.326+0000 7f8a21ffb640 10 client.?.objecter ms_handle_connect 0x7f8a2c0ae6e0 2024-03-25T18:10:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.326+0000 7f8a21ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:07.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.331+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2967425042 0 0) 0x7f8a2c1d2e60 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.331+0000 7f8a21ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:07.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.331+0000 7f8a21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:07.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.331+0000 7f8a21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:07.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.331+0000 7f8a21ffb640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:10:07.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.331+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a10004eb0 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.333+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 830047609 0 0) 0x7f8a2c0abb60 con 0x7f8a2c0704e0 2024-03-25T18:10:07.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.333+0000 7f8a21ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:07.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.333+0000 7f8a21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:07.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.333+0000 7f8a21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:07.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.333+0000 7f8a21ffb640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:10:07.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.333+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a2c1d2e60 con 0x7f8a2c0704e0 2024-03-25T18:10:07.350 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.350+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168874138 0 0) 0x7f8a2c0ac080 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.350+0000 7f8a21ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:07.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.350+0000 7f8a21ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:07.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.350+0000 7f8a21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:07.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.350+0000 7f8a21ffb640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:10:07.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.350+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8a2c0abb60 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.357+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3340220058 0 0) 0x7f8a2c1d2e60 con 0x7f8a2c0704e0 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.357+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a2c0ac080 con 0x7f8a2c0704e0 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.357+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8a28001bc0 con 0x7f8a2c0704e0 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.357+0000 7f8a21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.357+0000 7f8a21ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.357+0000 7f8a21ffb640 10 monclient(hunting): dump: 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:07.359 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.357+0000 7f8a21ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.358+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3781717590 0 0) 0x7f8a10004eb0 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.358+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a2c1d2e60 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.358+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8a24001cb0 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.358+0000 7f8a21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.358+0000 7f8a21ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.358+0000 7f8a21ffb640 10 monclient(hunting): dump: 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:07.360 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:07.361 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:07.361 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:07.361 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:07.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.363+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4274315905 0 0) 0x7f8a2c0abb60 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.364+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8a10004eb0 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.364+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8a1c001860 con 0x7f8a2c0ae6e0 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.364+0000 7f8a21ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.364+0000 7f8a21ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.364+0000 7f8a21ffb640 10 monclient(hunting): dump: 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:07.365 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:07.366 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:07.366 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:07.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.365+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3728119990 0 0) 0x7f8a2c1d2e60 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.365+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 >> v1:172.21.15.37:6790/0 conn(0x7f8a2c0ae6e0 legacy=0x7f8a2c136c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:07.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 >> v1:172.21.15.37:6789/0 conn(0x7f8a2c0704e0 legacy=0x7f8a2c1d5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:07.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a21ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:07.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a21ffb640 1 monclient: found mon.b 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a21ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a2c0ab780 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a21ffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a21ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a33625640 5 monclient: authenticate success, global_id 4327 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a33625640 10 monclient: _renew_subs 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a33625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a33625640 1 -- 172.21.15.37:0/3933249389 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a2c142800 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a33625640 10 client.4327.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a33625640 10 monclient: _renew_subs 2024-03-25T18:10:07.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a33625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:07.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a33625640 1 -- 172.21.15.37:0/3933249389 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a2c137380 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.366+0000 7f8a33625640 1 librados: init done 2024-03-25T18:10:07.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.367+0000 7f8a33625640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:07.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.367+0000 7f8a33625640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:07.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.367+0000 7f8a33625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:07.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.367+0000 7f8a33625640 1 -- 172.21.15.37:0/3933249389 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a2c0ab9d0 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.369+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8a24003990 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.369+0000 7f8a21ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:07.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.369+0000 7f8a237fe640 4 set_mon_vals no callback set 2024-03-25T18:10:07.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.386+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8a24003ca0 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.386+0000 7f8a21ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.386+0000 7f8a21ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.386+0000 7f8a21ffb640 10 monclient: dump: 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:07.387 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:07.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.386+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f8a24019c70 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.387+0000 7f8a313c1640 10 monclient: get_auth_request con 0x7f8a1006e160 auth_method 0 2024-03-25T18:10:07.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.387+0000 7f8a21ffb640 10 client.4327.objecter ms_handle_connect 0x7f8a1006e160 2024-03-25T18:10:07.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.390+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f8a24053940 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.390+0000 7f8a21ffb640 10 client.4327.objecter ms_dispatch 0x7f8a2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:07.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.390+0000 7f8a21ffb640 3 client.4327.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:07.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.390+0000 7f8a21ffb640 3 client.4327.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:07.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.390+0000 7f8a21ffb640 20 client.4327.objecter dump_active .. 0 homeless 2024-03-25T18:10:07.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.654+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8a2404f280 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.654+0000 7f8a21ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:07.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.654+0000 7f8a21ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:07.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.891+0000 7f8a33625640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:07.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.891+0000 7f8a33625640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:07.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.891+0000 7f8a33625640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:07.892 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.891+0000 7f8a33625640 1 -- 172.21.15.37:0/3933249389 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8a2c0abcf0 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.928+0000 7f8a21ffb640 1 -- 172.21.15.37:0/3933249389 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f8a24051380 con 0x7f8a2c1d5a30 2024-03-25T18:10:07.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.928+0000 7f8a21ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:07.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.928+0000 7f8a21ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:07.929 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:10:07.929 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:07.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.930+0000 7f8a33625640 10 librados: watch_flush enter 2024-03-25T18:10:07.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.930+0000 7f8a33625640 10 librados: watch_flush exit 2024-03-25T18:10:07.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.930+0000 7f8a33625640 20 client.4327.objecter shutdown clearing up homeless session... 2024-03-25T18:10:07.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.930+0000 7f8a33625640 10 client.4327.objecter successfully canceled tick 2024-03-25T18:10:07.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.931+0000 7f8a33625640 1 -- 172.21.15.37:0/3933249389 >> v1:172.21.15.117:6800/3703993921 conn(0x7f8a1006e160 legacy=0x7f8a10070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:07.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.931+0000 7f8a33625640 10 monclient: shutdown 2024-03-25T18:10:07.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.931+0000 7f8a33625640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:07.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.931+0000 7f8a33625640 1 -- 172.21.15.37:0/3933249389 >> v1:172.21.15.117:6789/0 conn(0x7f8a2c1d5a30 legacy=0x7f8a2c1e0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:07.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.931+0000 7f8a33625640 1 -- 172.21.15.37:0/3933249389 shutdown_connections 2024-03-25T18:10:07.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.931+0000 7f8a33625640 1 -- 172.21.15.37:0/3933249389 >> 172.21.15.37:0/3933249389 conn(0x7f8a2c06b630 msgr2=0x7f8a2c13c0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:07.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.931+0000 7f8a33625640 1 -- 172.21.15.37:0/3933249389 shutdown_connections 2024-03-25T18:10:07.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.931+0000 7f8a33625640 1 -- 172.21.15.37:0/3933249389 wait complete. 2024-03-25T18:10:07.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:07.931+0000 7f8a33625640 1 librados: shutdown 2024-03-25T18:10:07.943 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:14.750 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:14.865 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.865+0000 7f539a9e8640 10 monclient: get_monmap_and_config 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.865+0000 7f539a9e8640 10 monclient: build_initial_monmap 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.865+0000 7f539a9e8640 10 monclient: monmap: 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:10:14.865988+0000 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:10:14.865988+0000 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:10:14.866 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:14.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.865+0000 7f539a9e8640 1 Processor -- start 2024-03-25T18:10:14.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.865+0000 7f539a9e8640 1 -- start start 2024-03-25T18:10:14.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.865+0000 7f539a9e8640 10 monclient: init 2024-03-25T18:10:14.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.866+0000 7f539a9e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:14.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.866+0000 7f539a9e8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:14.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.866+0000 7f539a9e8640 10 monclient(hunting): picked mon.noname-a con 0x7f53940b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:14.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.866+0000 7f539a9e8640 10 monclient(hunting): picked mon.noname-b con 0x7f53940b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:14.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.866+0000 7f539a9e8640 10 monclient(hunting): picked mon.noname-c con 0x7f539414be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:14.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.866+0000 7f539a9e8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53940af940 con 0x7f53940b4d60 2024-03-25T18:10:14.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.866+0000 7f539a9e8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53940b0a70 con 0x7f539414be90 2024-03-25T18:10:14.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.866+0000 7f539a9e8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53940b0d50 con 0x7f53940b1730 2024-03-25T18:10:14.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.866+0000 7f539a9e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:14.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.866+0000 7f539a9e8640 10 monclient(hunting): authenticate will time out at 6970.118968s 2024-03-25T18:10:14.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.867+0000 7f5393fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f53940b4d60 0x7f539413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47626/0 (socket says 172.21.15.37:47626) 2024-03-25T18:10:14.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.867+0000 7f5393fff640 1 -- 172.21.15.37:0/2293371120 learned_addr learned my addr 172.21.15.37:0/2293371120 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:14.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.889+0000 7f5392ffd640 1 -- 172.21.15.37:0/2293371120 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 306005123 0 0) 0x7f53940b0a70 con 0x7f539414be90 2024-03-25T18:10:14.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.889+0000 7f5392ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:14.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.889+0000 7f5392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:14.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.889+0000 7f5392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:14.890 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.889+0000 7f5392ffd640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:10:14.891 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.889+0000 7f5392ffd640 1 -- 172.21.15.37:0/2293371120 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f537c003520 con 0x7f539414be90 2024-03-25T18:10:14.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.898+0000 7f5392ffd640 1 -- 172.21.15.37:0/2293371120 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533211277 0 0) 0x7f53940af940 con 0x7f53940b4d60 2024-03-25T18:10:14.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.898+0000 7f5392ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:14.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.898+0000 7f5392ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:14.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.898+0000 7f5392ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:14.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.898+0000 7f5392ffd640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:10:14.899 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.898+0000 7f5392ffd640 1 -- 172.21.15.37:0/2293371120 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53940b0a70 con 0x7f53940b4d60 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 1 -- 172.21.15.37:0/2293371120 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 48737491 0 0) 0x7f537c003520 con 0x7f539414be90 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 1 -- 172.21.15.37:0/2293371120 >> v1:172.21.15.37:6790/0 conn(0x7f53940b1730 legacy=0x7f53940b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 1 -- 172.21.15.37:0/2293371120 >> v1:172.21.15.37:6789/0 conn(0x7f53940b4d60 legacy=0x7f539413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 1 monclient: found mon.noname-c 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 1 -- 172.21.15.37:0/2293371120 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53940b1030 con 0x7f539414be90 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 1 -- 172.21.15.37:0/2293371120 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f538c001a10 con 0x7f539414be90 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:14.901 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f5392ffd640 10 monclient: dump: 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f539a9e8640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:10:14.902 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.899+0000 7f539a9e8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:14.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.933+0000 7f5392ffd640 1 -- 172.21.15.37:0/2293371120 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f538c001b70 con 0x7f539414be90 2024-03-25T18:10:14.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.933+0000 7f5392ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:14.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.933+0000 7f5392ffd640 1 -- 172.21.15.37:0/2293371120 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f538c001f80 con 0x7f539414be90 2024-03-25T18:10:14.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.933+0000 7f5392ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:14.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.933+0000 7f5392ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:14.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.933+0000 7f5392ffd640 10 monclient: dump: 2024-03-25T18:10:14.934 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:14.934 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:14.935 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:14.935 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:14.935 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:14.935 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:14.935 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:14.935 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:14.935 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:14.935 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:14.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.933+0000 7f539a9e8640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:14.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.933+0000 7f539a9e8640 4 set_mon_vals no callback set 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.934+0000 7f539a9e8640 10 monclient: shutdown 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.934+0000 7f539a9e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.934+0000 7f539a9e8640 1 -- 172.21.15.37:0/2293371120 >> v1:172.21.15.117:6789/0 conn(0x7f539414be90 legacy=0x7f539414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.934+0000 7f539a9e8640 1 -- 172.21.15.37:0/2293371120 shutdown_connections 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.934+0000 7f539a9e8640 1 -- 172.21.15.37:0/2293371120 >> 172.21.15.37:0/2293371120 conn(0x7f53940617a0 msgr2=0x7f5394061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.934+0000 7f539a9e8640 1 -- 172.21.15.37:0/2293371120 shutdown_connections 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.934+0000 7f539a9e8640 1 -- 172.21.15.37:0/2293371120 wait complete. 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.935+0000 7f539a9e8640 10 monclient: build_initial_monmap 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.935+0000 7f539a9e8640 10 monclient: monmap: 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:14.937 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.935+0000 7f539a9e8640 1 librados: starting msgr at 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.935+0000 7f539a9e8640 1 librados: starting objecter 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.935+0000 7f539a9e8640 1 Processor -- start 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 1 -- start start 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 1 librados: setting wanted keys 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 1 librados: calling monclient init 2024-03-25T18:10:14.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 10 monclient: init 2024-03-25T18:10:14.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:14.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:14.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 10 monclient(hunting): picked mon.noname-a con 0x7f53940b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:14.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 10 monclient(hunting): picked mon.noname-b con 0x7f53940b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:14.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 10 monclient(hunting): picked mon.noname-c con 0x7f539414be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:14.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53941363f0 con 0x7f53940b1730 2024-03-25T18:10:14.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f53941374e0 con 0x7f53940b4d60 2024-03-25T18:10:14.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.936+0000 7f539a9e8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5394138140 con 0x7f539414be90 2024-03-25T18:10:14.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.937+0000 7f539a9e8640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:14.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.937+0000 7f539a9e8640 10 monclient(hunting): authenticate will time out at 6970.189296s 2024-03-25T18:10:14.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.943+0000 7f5393fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f53940b1730 0x7f5394136dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47682/0 (socket says 172.21.15.37:47682) 2024-03-25T18:10:14.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.943+0000 7f5393fff640 1 -- 172.21.15.37:0/1516777104 learned_addr learned my addr 172.21.15.37:0/1516777104 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:14.945 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.944+0000 7f5390ff9640 10 client.?.objecter ms_handle_connect 0x7f53940b1730 2024-03-25T18:10:14.945 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.944+0000 7f5390ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:14.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.949+0000 7f5390ff9640 10 client.?.objecter ms_handle_connect 0x7f539414be90 2024-03-25T18:10:14.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.949+0000 7f5390ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:14.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.956+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2058796268 0 0) 0x7f53941363f0 con 0x7f53940b1730 2024-03-25T18:10:14.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.956+0000 7f5390ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:14.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.956+0000 7f5390ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:14.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.956+0000 7f5390ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:14.958 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.956+0000 7f5390ff9640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:10:14.958 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.956+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5380004eb0 con 0x7f53940b1730 2024-03-25T18:10:14.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.958+0000 7f5390ff9640 10 client.?.objecter ms_handle_connect 0x7f53940b4d60 2024-03-25T18:10:14.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.958+0000 7f5390ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:14.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.963+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291712206 0 0) 0x7f5394138140 con 0x7f539414be90 2024-03-25T18:10:14.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.963+0000 7f5390ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:14.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.963+0000 7f5390ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:14.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.963+0000 7f5390ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:14.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.963+0000 7f5390ff9640 10 monclient(hunting): my global_id is 4328 2024-03-25T18:10:14.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.963+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53941363f0 con 0x7f539414be90 2024-03-25T18:10:14.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.966+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336598360 0 0) 0x7f53941374e0 con 0x7f53940b4d60 2024-03-25T18:10:14.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.966+0000 7f5390ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:10:14.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.966+0000 7f5390ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:14.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.966+0000 7f5390ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:14.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.966+0000 7f5390ff9640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:10:14.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.966+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5394138140 con 0x7f53940b4d60 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.971+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3642964237 0 0) 0x7f5380004eb0 con 0x7f53940b1730 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.972+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53941374e0 con 0x7f53940b1730 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.972+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f53840019b0 con 0x7f53940b1730 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.972+0000 7f5390ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.972+0000 7f5390ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.972+0000 7f5390ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:14.973 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:14.974 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:14.974 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:14.974 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.972+0000 7f5390ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:14.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.983+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3695929706 0 0) 0x7f53941363f0 con 0x7f539414be90 2024-03-25T18:10:14.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.983+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5380004eb0 con 0x7f539414be90 2024-03-25T18:10:14.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.983+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f538c001e80 con 0x7f539414be90 2024-03-25T18:10:14.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.983+0000 7f5390ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.983+0000 7f5390ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.983+0000 7f5390ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:14.985 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:14.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.985+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3930488683 0 0) 0x7f5394138140 con 0x7f53940b4d60 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.985+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f53941363f0 con 0x7f53940b4d60 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.985+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5388001860 con 0x7f53940b4d60 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.985+0000 7f5390ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.985+0000 7f5390ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.985+0000 7f5390ff9640 10 monclient(hunting): dump: 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:14.987 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:14.988 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:14.988 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:14.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2294725526 0 0) 0x7f53941374e0 con 0x7f53940b1730 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 >> v1:172.21.15.37:6790/0 conn(0x7f539414be90 legacy=0x7f5394143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 >> v1:172.21.15.117:6789/0 conn(0x7f53940b4d60 legacy=0x7f539412fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f5390ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f5390ff9640 1 monclient: found mon.a 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f5390ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f5390ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5394139090 con 0x7f53940b1730 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f5390ff9640 10 monclient: _finish_auth 0 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f5390ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f5390ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f539a9e8640 5 monclient: authenticate success, global_id 4332 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f539a9e8640 10 monclient: _renew_subs 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f539a9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:10:14.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f539a9e8640 1 -- 172.21.15.37:0/1516777104 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5394136030 con 0x7f53940b1730 2024-03-25T18:10:15.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f539a9e8640 10 client.4332.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:15.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f539a9e8640 10 monclient: _renew_subs 2024-03-25T18:10:15.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f539a9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:10:15.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f539a9e8640 1 -- 172.21.15.37:0/1516777104 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5394130650 con 0x7f53940b1730 2024-03-25T18:10:15.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.997+0000 7f539a9e8640 1 librados: init done 2024-03-25T18:10:15.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.998+0000 7f539a9e8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:15.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.998+0000 7f539a9e8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:15.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.998+0000 7f539a9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:10:15.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:14.998+0000 7f539a9e8640 1 -- 172.21.15.37:0/1516777104 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53940ae6c0 con 0x7f53940b1730 2024-03-25T18:10:15.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.331+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f53840047e0 con 0x7f53940b1730 2024-03-25T18:10:15.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.331+0000 7f5390ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:15.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.331+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5384004af0 con 0x7f53940b1730 2024-03-25T18:10:15.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.331+0000 7f5390ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:15.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.331+0000 7f5390ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:15.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.331+0000 7f5390ff9640 10 monclient: dump: 2024-03-25T18:10:15.333 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:15.333 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:15.333 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:15.333 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:15.333 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:15.333 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:15.333 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:15.333 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:15.333 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:15.333 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:15.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.331+0000 7f53927fc640 4 set_mon_vals no callback set 2024-03-25T18:10:15.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.332+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f538401ab00 con 0x7f53940b1730 2024-03-25T18:10:15.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.332+0000 7f53937fe640 10 monclient: get_auth_request con 0x7f538006e0f0 auth_method 0 2024-03-25T18:10:15.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.333+0000 7f5390ff9640 10 client.4332.objecter ms_handle_connect 0x7f538006e0f0 2024-03-25T18:10:15.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.338+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f5384054850 con 0x7f53940b1730 2024-03-25T18:10:15.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.338+0000 7f5390ff9640 10 client.4332.objecter ms_dispatch 0x7f5394000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:15.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.338+0000 7f5390ff9640 3 client.4332.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:15.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.338+0000 7f5390ff9640 3 client.4332.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:15.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.338+0000 7f5390ff9640 20 client.4332.objecter dump_active .. 0 homeless 2024-03-25T18:10:15.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.338+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5384050190 con 0x7f53940b1730 2024-03-25T18:10:15.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.338+0000 7f5390ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:15.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.338+0000 7f5390ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.565+0000 7f539a9e8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.565+0000 7f539a9e8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.565+0000 7f539a9e8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:10:15.566 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.565+0000 7f539a9e8640 1 -- 172.21.15.37:0/1516777104 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f53940afbe0 con 0x7f53940b1730 2024-03-25T18:10:15.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.602+0000 7f5390ff9640 1 -- 172.21.15.37:0/1516777104 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f5384052290 con 0x7f53940b1730 2024-03-25T18:10:15.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.602+0000 7f5390ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:15.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.602+0000 7f5390ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:15.603 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:10:15.603 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:15.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 10 librados: watch_flush enter 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 10 librados: watch_flush exit 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 20 client.4332.objecter shutdown clearing up homeless session... 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 10 client.4332.objecter successfully canceled tick 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 1 -- 172.21.15.37:0/1516777104 >> v1:172.21.15.117:6800/3703993921 conn(0x7f538006e0f0 legacy=0x7f53800705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 10 monclient: shutdown 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 1 -- 172.21.15.37:0/1516777104 >> v1:172.21.15.37:6789/0 conn(0x7f53940b1730 legacy=0x7f5394136dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 1 -- 172.21.15.37:0/1516777104 shutdown_connections 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 1 -- 172.21.15.37:0/1516777104 >> 172.21.15.37:0/1516777104 conn(0x7f53940617a0 msgr2=0x7f539413c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 1 -- 172.21.15.37:0/1516777104 shutdown_connections 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 1 -- 172.21.15.37:0/1516777104 wait complete. 2024-03-25T18:10:15.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:15.604+0000 7f539a9e8640 1 librados: shutdown 2024-03-25T18:10:15.618 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:22.424 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:22.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.540+0000 7f801d0b3640 10 monclient: get_monmap_and_config 2024-03-25T18:10:22.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.540+0000 7f801d0b3640 10 monclient: build_initial_monmap 2024-03-25T18:10:22.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.540+0000 7f801d0b3640 10 monclient: monmap: 2024-03-25T18:10:22.541 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:22.541 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:22.541 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:10:22.541356+0000 2024-03-25T18:10:22.541 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:10:22.541356+0000 2024-03-25T18:10:22.541 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:22.541 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:22.542 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:22.542 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:10:22.542 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:10:22.542 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.541+0000 7f801d0b3640 1 Processor -- start 2024-03-25T18:10:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.541+0000 7f801d0b3640 1 -- start start 2024-03-25T18:10:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.541+0000 7f801d0b3640 10 monclient: init 2024-03-25T18:10:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.541+0000 7f801d0b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.541+0000 7f801d0b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.541+0000 7f801d0b3640 10 monclient(hunting): picked mon.noname-b con 0x7f80180a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.541+0000 7f801d0b3640 10 monclient(hunting): picked mon.noname-c con 0x7f801813f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.541+0000 7f801d0b3640 10 monclient(hunting): picked mon.noname-a con 0x7f80181430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.542+0000 7f801d0b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8018142520 con 0x7f80181430c0 2024-03-25T18:10:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.542+0000 7f801d0b3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80180b28a0 con 0x7f801813f710 2024-03-25T18:10:22.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.542+0000 7f801d0b3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80180b2a30 con 0x7f80180a98f0 2024-03-25T18:10:22.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.542+0000 7f801d0b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:22.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.542+0000 7f801d0b3640 10 monclient(hunting): authenticate will time out at 6977.794347s 2024-03-25T18:10:22.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.550+0000 7f801759e640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f80181430c0 0x7f801814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46908/0 (socket says 172.21.15.37:46908) 2024-03-25T18:10:22.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.550+0000 7f801759e640 1 -- 172.21.15.37:0/1511244015 learned_addr learned my addr 172.21.15.37:0/1511244015 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:22.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.567+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 896731568 0 0) 0x7f80180b28a0 con 0x7f801813f710 2024-03-25T18:10:22.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.567+0000 7f8015d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:10:22.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.567+0000 7f8015d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:22.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.567+0000 7f8015d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:22.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.567+0000 7f8015d9b640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:10:22.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.567+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8000003520 con 0x7f801813f710 2024-03-25T18:10:22.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.577+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1038896379 0 0) 0x7f8018142520 con 0x7f80181430c0 2024-03-25T18:10:22.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.577+0000 7f8015d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:10:22.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.577+0000 7f8015d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:22.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.577+0000 7f8015d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:22.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.577+0000 7f8015d9b640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:10:22.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.577+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80180b28a0 con 0x7f80181430c0 2024-03-25T18:10:22.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 580227380 0 0) 0x7f80180b2a30 con 0x7f80180a98f0 2024-03-25T18:10:22.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:10:22.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:22.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:22.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8018142520 con 0x7f80180a98f0 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 890616814 0 0) 0x7f80180b28a0 con 0x7f80181430c0 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 >> v1:172.21.15.37:6790/0 conn(0x7f80180a98f0 legacy=0x7f80180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 >> v1:172.21.15.117:6789/0 conn(0x7f801813f710 legacy=0x7f80181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 1 monclient: found mon.noname-a 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80180b44c0 con 0x7f80181430c0 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 10 monclient: _finish_auth 0 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f80100019f0 con 0x7f80181430c0 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:22.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f8015d9b640 10 monclient: dump: 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f801d0b3640 5 monclient: authenticate success, global_id 4335 2024-03-25T18:10:22.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.590+0000 7f801d0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:22.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.617+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8010001b50 con 0x7f80181430c0 2024-03-25T18:10:22.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.617+0000 7f8015d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:22.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.617+0000 7f8015d9b640 1 -- 172.21.15.37:0/1511244015 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8010001f60 con 0x7f80181430c0 2024-03-25T18:10:22.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.617+0000 7f801d0b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:22.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.617+0000 7f801d0b3640 4 set_mon_vals no callback set 2024-03-25T18:10:22.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.617+0000 7f8015d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.617+0000 7f8015d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.617+0000 7f8015d9b640 10 monclient: dump: 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:22.619 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:22.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.618+0000 7f801d0b3640 10 monclient: shutdown 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.618+0000 7f801d0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.618+0000 7f801d0b3640 1 -- 172.21.15.37:0/1511244015 >> v1:172.21.15.37:6789/0 conn(0x7f80181430c0 legacy=0x7f801814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.618+0000 7f801d0b3640 1 -- 172.21.15.37:0/1511244015 shutdown_connections 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.618+0000 7f801d0b3640 1 -- 172.21.15.37:0/1511244015 >> 172.21.15.37:0/1511244015 conn(0x7f80180b7ea0 msgr2=0x7f80180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.618+0000 7f801d0b3640 1 -- 172.21.15.37:0/1511244015 shutdown_connections 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.618+0000 7f801d0b3640 1 -- 172.21.15.37:0/1511244015 wait complete. 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.619+0000 7f801d0b3640 10 monclient: build_initial_monmap 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.619+0000 7f801d0b3640 10 monclient: monmap: 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:10:22.622 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.619+0000 7f801d0b3640 1 librados: starting msgr at 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.619+0000 7f801d0b3640 1 librados: starting objecter 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 1 Processor -- start 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 1 -- start start 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 1 librados: setting wanted keys 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 1 librados: calling monclient init 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 10 monclient: init 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:22.623 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:10:22.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 10 monclient(hunting): picked mon.noname-b con 0x7f80180a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:22.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 10 monclient(hunting): picked mon.noname-c con 0x7f801813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:22.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 10 monclient(hunting): picked mon.noname-a con 0x7f80181430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:22.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80181d48f0 con 0x7f80181430c0 2024-03-25T18:10:22.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80181397d0 con 0x7f80180a98f0 2024-03-25T18:10:22.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80181d4b50 con 0x7f801813f710 2024-03-25T18:10:22.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:22.624 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.620+0000 7f801d0b3640 10 monclient(hunting): authenticate will time out at 6977.873248s 2024-03-25T18:10:22.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.627+0000 7f801759e640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f80181430c0 0x7f80181d7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46970/0 (socket says 172.21.15.37:46970) 2024-03-25T18:10:22.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.627+0000 7f801759e640 1 -- 172.21.15.37:0/3345204191 learned_addr learned my addr 172.21.15.37:0/3345204191 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:22.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.628+0000 7f7fff7fe640 10 client.?.objecter ms_handle_connect 0x7f80181430c0 2024-03-25T18:10:22.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.628+0000 7f7fff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:22.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.632+0000 7f7fff7fe640 10 client.?.objecter ms_handle_connect 0x7f801813f710 2024-03-25T18:10:22.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.632+0000 7f7fff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:22.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.634+0000 7f7fff7fe640 10 client.?.objecter ms_handle_connect 0x7f80180a98f0 2024-03-25T18:10:22.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.634+0000 7f7fff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:22.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.642+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085016843 0 0) 0x7f80181d4b50 con 0x7f801813f710 2024-03-25T18:10:22.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.642+0000 7f7fff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:22.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.642+0000 7f7fff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:22.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.642+0000 7f7fff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:22.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.642+0000 7f7fff7fe640 10 monclient(hunting): my global_id is 4334 2024-03-25T18:10:22.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.642+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8004004eb0 con 0x7f801813f710 2024-03-25T18:10:22.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.644+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3382527453 0 0) 0x7f80181397d0 con 0x7f80180a98f0 2024-03-25T18:10:22.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.644+0000 7f7fff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:22.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.644+0000 7f7fff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:22.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.644+0000 7f7fff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:22.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.644+0000 7f7fff7fe640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:10:22.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.644+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80181d4b50 con 0x7f80180a98f0 2024-03-25T18:10:22.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.649+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2653404285 0 0) 0x7f80181d48f0 con 0x7f80181430c0 2024-03-25T18:10:22.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.649+0000 7f7fff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:22.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.649+0000 7f7fff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:22.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.649+0000 7f7fff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:22.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.649+0000 7f7fff7fe640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:10:22.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.649+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80181397d0 con 0x7f80181430c0 2024-03-25T18:10:22.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.657+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4133484871 0 0) 0x7f8004004eb0 con 0x7f801813f710 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.657+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80181d48f0 con 0x7f801813f710 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.657+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f800c001950 con 0x7f801813f710 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.657+0000 7f7fff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.657+0000 7f7fff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.657+0000 7f7fff7fe640 10 monclient(hunting): dump: 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:22.658 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:22.659 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:22.659 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:22.659 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.657+0000 7f7fff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:22.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.668+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3752463428 0 0) 0x7f80181d4b50 con 0x7f80180a98f0 2024-03-25T18:10:22.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.668+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8004004eb0 con 0x7f80180a98f0 2024-03-25T18:10:22.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.668+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8008001870 con 0x7f80180a98f0 2024-03-25T18:10:22.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.668+0000 7f7fff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.668+0000 7f7fff7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.668+0000 7f7fff7fe640 10 monclient(hunting): dump: 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:22.670 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4018014612 0 0) 0x7f80181d48f0 con 0x7f801813f710 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 >> v1:172.21.15.117:6789/0 conn(0x7f80180a98f0 legacy=0x7f80181d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 >> v1:172.21.15.37:6789/0 conn(0x7f80181430c0 legacy=0x7f80181d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f7fff7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f7fff7fe640 1 monclient: found mon.c 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f7fff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f7fff7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80180b16b0 con 0x7f801813f710 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f7fff7fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f7fff7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f7fff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:22.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f801d0b3640 5 monclient: authenticate success, global_id 4334 2024-03-25T18:10:22.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f801d0b3640 10 monclient: _renew_subs 2024-03-25T18:10:22.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f801d0b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:10:22.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.671+0000 7f801d0b3640 1 -- 172.21.15.37:0/3345204191 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80180b0c10 con 0x7f801813f710 2024-03-25T18:10:22.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.672+0000 7f801d0b3640 10 client.4334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:22.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.672+0000 7f801d0b3640 10 monclient: _renew_subs 2024-03-25T18:10:22.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.672+0000 7f801d0b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:10:22.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.672+0000 7f801d0b3640 1 -- 172.21.15.37:0/3345204191 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80180b0610 con 0x7f801813f710 2024-03-25T18:10:22.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.672+0000 7f801d0b3640 1 librados: init done 2024-03-25T18:10:22.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.673+0000 7f801d0b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:22.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.673+0000 7f801d0b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:22.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.673+0000 7f801d0b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:10:22.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:22.673+0000 7f801d0b3640 1 -- 172.21.15.37:0/3345204191 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80181d4ce0 con 0x7f801813f710 2024-03-25T18:10:23.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.002+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f800c003610 con 0x7f801813f710 2024-03-25T18:10:23.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.002+0000 7f7fff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:23.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.002+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f800c001d80 con 0x7f801813f710 2024-03-25T18:10:23.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.002+0000 7f7fff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:23.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.002+0000 7f7fff7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:23.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.002+0000 7f7fff7fe640 10 monclient: dump: 2024-03-25T18:10:23.003 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:23.003 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:23.003 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:23.003 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:23.004 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:23.004 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:23.004 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:23.004 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:23.004 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:23.004 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:23.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.002+0000 7f801559a640 4 set_mon_vals no callback set 2024-03-25T18:10:23.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.002+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f800c0197d0 con 0x7f801813f710 2024-03-25T18:10:23.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.003+0000 7f8016d9d640 10 monclient: get_auth_request con 0x7f800406df20 auth_method 0 2024-03-25T18:10:23.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.003+0000 7f7fff7fe640 10 client.4334.objecter ms_handle_connect 0x7f800406df20 2024-03-25T18:10:23.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.009+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f800c053550 con 0x7f801813f710 2024-03-25T18:10:23.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.009+0000 7f7fff7fe640 10 client.4334.objecter ms_dispatch 0x7f8018000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:23.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.009+0000 7f7fff7fe640 3 client.4334.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:23.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.009+0000 7f7fff7fe640 3 client.4334.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:23.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.009+0000 7f7fff7fe640 20 client.4334.objecter dump_active .. 0 homeless 2024-03-25T18:10:23.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.009+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f800c04ee90 con 0x7f801813f710 2024-03-25T18:10:23.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.009+0000 7f7fff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:23.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.009+0000 7f7fff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:23.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.225+0000 7f801d0b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:23.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.225+0000 7f801d0b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:23.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.225+0000 7f801d0b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:10:23.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.225+0000 7f801d0b3640 1 -- 172.21.15.37:0/3345204191 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80180627b0 con 0x7f801813f710 2024-03-25T18:10:23.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.256+0000 7f7fff7fe640 1 -- 172.21.15.37:0/3345204191 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f800c050f90 con 0x7f801813f710 2024-03-25T18:10:23.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.256+0000 7f7fff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:23.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.256+0000 7f7fff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:23.257 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:10:23.257 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:23.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 10 librados: watch_flush enter 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 10 librados: watch_flush exit 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 20 client.4334.objecter shutdown clearing up homeless session... 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 10 client.4334.objecter successfully canceled tick 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 1 -- 172.21.15.37:0/3345204191 >> v1:172.21.15.117:6800/3703993921 conn(0x7f800406df20 legacy=0x7f80040703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 10 monclient: shutdown 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 1 -- 172.21.15.37:0/3345204191 >> v1:172.21.15.37:6790/0 conn(0x7f801813f710 legacy=0x7f80180aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 1 -- 172.21.15.37:0/3345204191 shutdown_connections 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 1 -- 172.21.15.37:0/3345204191 >> 172.21.15.37:0/3345204191 conn(0x7f80180b7ea0 msgr2=0x7f8018074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 1 -- 172.21.15.37:0/3345204191 shutdown_connections 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 1 -- 172.21.15.37:0/3345204191 wait complete. 2024-03-25T18:10:23.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:23.258+0000 7f801d0b3640 1 librados: shutdown 2024-03-25T18:10:23.271 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:24.398 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:02:59.744 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:24.448 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:02:59.793 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:24.494 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:02:59.840 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:24.495 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:02:59.808 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:24.502 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:02:59.848 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:24.506 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:02:59.819 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:24.511 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:02:59.857 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:24.529 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:02:59.842 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:24.547 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:02:59.893 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:30.077 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:30.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.193+0000 7fbe6100f640 10 monclient: get_monmap_and_config 2024-03-25T18:10:30.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.193+0000 7fbe6100f640 10 monclient: build_initial_monmap 2024-03-25T18:10:30.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.193+0000 7fbe6100f640 10 monclient: monmap: 2024-03-25T18:10:30.194 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:30.194 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:30.194 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:10:30.194379+0000 2024-03-25T18:10:30.194 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:10:30.194379+0000 2024-03-25T18:10:30.195 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:30.195 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:30.195 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:30.195 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:10:30.195 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:10:30.195 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:30.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.194+0000 7fbe6100f640 1 Processor -- start 2024-03-25T18:10:30.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.194+0000 7fbe6100f640 1 -- start start 2024-03-25T18:10:30.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.194+0000 7fbe6100f640 10 monclient: init 2024-03-25T18:10:30.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.194+0000 7fbe6100f640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:30.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.194+0000 7fbe6100f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:30.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.194+0000 7fbe6100f640 10 monclient(hunting): picked mon.noname-a con 0x7fbe5c0b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.194+0000 7fbe6100f640 10 monclient(hunting): picked mon.noname-c con 0x7fbe5c0b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.194+0000 7fbe6100f640 10 monclient(hunting): picked mon.noname-b con 0x7fbe5c14be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.195+0000 7fbe6100f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c0af940 con 0x7fbe5c0b4d60 2024-03-25T18:10:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.195+0000 7fbe6100f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c0b0a70 con 0x7fbe5c0b1730 2024-03-25T18:10:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.195+0000 7fbe6100f640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c0b0d50 con 0x7fbe5c14be90 2024-03-25T18:10:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.195+0000 7fbe6100f640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.195+0000 7fbe6100f640 10 monclient(hunting): authenticate will time out at 6985.447420s 2024-03-25T18:10:30.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.205+0000 7fbe53fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fbe5c0b1730 0x7fbe5c0b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:59354/0 (socket says 172.21.15.37:59354) 2024-03-25T18:10:30.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.205+0000 7fbe53fff640 1 -- 172.21.15.37:0/4239025179 learned_addr learned my addr 172.21.15.37:0/4239025179 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:30.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 1 -- 172.21.15.37:0/4239025179 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3439215098 0 0) 0x7fbe5c0b0a70 con 0x7fbe5c0b1730 2024-03-25T18:10:30.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:10:30.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:30.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:30.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:10:30.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 1 -- 172.21.15.37:0/4239025179 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe48003520 con 0x7fbe5c0b1730 2024-03-25T18:10:30.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 1 -- 172.21.15.37:0/4239025179 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3808466264 0 0) 0x7fbe5c0af940 con 0x7fbe5c0b4d60 2024-03-25T18:10:30.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 10 monclient(hunting): _negotiate 2024-03-25T18:10:30.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:30.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:30.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 10 monclient(hunting): my global_id is 4341 2024-03-25T18:10:30.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.216+0000 7fbe5a59c640 1 -- 172.21.15.37:0/4239025179 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe5c0b0a70 con 0x7fbe5c0b4d60 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.224+0000 7fbe5a59c640 1 -- 172.21.15.37:0/4239025179 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 179081113 0 0) 0x7fbe48003520 con 0x7fbe5c0b1730 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.224+0000 7fbe5a59c640 1 -- 172.21.15.37:0/4239025179 >> v1:172.21.15.37:6790/0 conn(0x7fbe5c14be90 legacy=0x7fbe5c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.224+0000 7fbe5a59c640 1 -- 172.21.15.37:0/4239025179 >> v1:172.21.15.37:6789/0 conn(0x7fbe5c0b4d60 legacy=0x7fbe5c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.224+0000 7fbe5a59c640 10 monclient: _finish_hunting 0 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.224+0000 7fbe5a59c640 1 monclient: found mon.noname-c 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.224+0000 7fbe5a59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe5a59c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe5a59c640 1 -- 172.21.15.37:0/4239025179 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c0b1030 con 0x7fbe5c0b1730 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe5a59c640 10 monclient: _finish_auth 0 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe5a59c640 10 monclient: _check_auth_tickets 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe5a59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:30.226 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe5a59c640 1 -- 172.21.15.37:0/4239025179 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbe44001690 con 0x7fbe5c0b1730 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe6100f640 5 monclient: authenticate success, global_id 4342 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe6100f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe5a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe5a59c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe5a59c640 10 monclient: dump: 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:30.227 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.225+0000 7fbe6100f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:30.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.242+0000 7fbe5a59c640 1 -- 172.21.15.37:0/4239025179 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe440017f0 con 0x7fbe5c0b1730 2024-03-25T18:10:30.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.242+0000 7fbe5a59c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:30.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.242+0000 7fbe5a59c640 1 -- 172.21.15.37:0/4239025179 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbe44001c00 con 0x7fbe5c0b1730 2024-03-25T18:10:30.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.242+0000 7fbe6100f640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:30.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.242+0000 7fbe6100f640 4 set_mon_vals no callback set 2024-03-25T18:10:30.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.242+0000 7fbe5a59c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:30.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.242+0000 7fbe5a59c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:30.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.242+0000 7fbe5a59c640 10 monclient: dump: 2024-03-25T18:10:30.244 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:30.244 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:30.244 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:30.244 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:30.244 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:30.244 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:30.244 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:30.244 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:30.244 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:30.244 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:30.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.243+0000 7fbe6100f640 10 monclient: shutdown 2024-03-25T18:10:30.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.243+0000 7fbe6100f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:30.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.243+0000 7fbe6100f640 1 -- 172.21.15.37:0/4239025179 >> v1:172.21.15.117:6789/0 conn(0x7fbe5c0b1730 legacy=0x7fbe5c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:30.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.243+0000 7fbe6100f640 1 -- 172.21.15.37:0/4239025179 shutdown_connections 2024-03-25T18:10:30.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.243+0000 7fbe6100f640 1 -- 172.21.15.37:0/4239025179 >> 172.21.15.37:0/4239025179 conn(0x7fbe5c0617a0 msgr2=0x7fbe5c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:30.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.243+0000 7fbe6100f640 1 -- 172.21.15.37:0/4239025179 shutdown_connections 2024-03-25T18:10:30.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.243+0000 7fbe6100f640 1 -- 172.21.15.37:0/4239025179 wait complete. 2024-03-25T18:10:30.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.244+0000 7fbe6100f640 10 monclient: build_initial_monmap 2024-03-25T18:10:30.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.244+0000 7fbe6100f640 10 monclient: monmap: 2024-03-25T18:10:30.247 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:30.247 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.244+0000 7fbe6100f640 1 librados: starting msgr at 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.244+0000 7fbe6100f640 1 librados: starting objecter 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.244+0000 7fbe6100f640 1 Processor -- start 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.244+0000 7fbe6100f640 1 -- start start 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.244+0000 7fbe6100f640 1 librados: setting wanted keys 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.244+0000 7fbe6100f640 1 librados: calling monclient init 2024-03-25T18:10:30.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.244+0000 7fbe6100f640 10 monclient: init 2024-03-25T18:10:30.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.245+0000 7fbe6100f640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:30.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.245+0000 7fbe6100f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:30.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.245+0000 7fbe6100f640 10 monclient(hunting): picked mon.noname-a con 0x7fbe5c0b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:30.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.245+0000 7fbe6100f640 10 monclient(hunting): picked mon.noname-b con 0x7fbe5c0b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:30.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.245+0000 7fbe6100f640 10 monclient(hunting): picked mon.noname-c con 0x7fbe5c14be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:30.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.245+0000 7fbe6100f640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c141790 con 0x7fbe5c0b1730 2024-03-25T18:10:30.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.245+0000 7fbe6100f640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c141a00 con 0x7fbe5c0b4d60 2024-03-25T18:10:30.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.245+0000 7fbe6100f640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe5c1d7a90 con 0x7fbe5c14be90 2024-03-25T18:10:30.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.245+0000 7fbe6100f640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:30.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.245+0000 7fbe6100f640 10 monclient(hunting): authenticate will time out at 6985.498144s 2024-03-25T18:10:30.251 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.251+0000 7fbe5b59e640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fbe5c14be90 0x7fbe5c1d7fb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:59548/0 (socket says 172.21.15.37:59548) 2024-03-25T18:10:30.252 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.251+0000 7fbe5b59e640 1 -- 172.21.15.37:0/3945085449 learned_addr learned my addr 172.21.15.37:0/3945085449 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:30.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.255+0000 7fbe537fe640 10 client.?.objecter ms_handle_connect 0x7fbe5c14be90 2024-03-25T18:10:30.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.255+0000 7fbe537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:30.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.262+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 473281190 0 0) 0x7fbe5c1d7a90 con 0x7fbe5c14be90 2024-03-25T18:10:30.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.262+0000 7fbe537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:30.263 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.262+0000 7fbe537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:30.263 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.262+0000 7fbe537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:30.263 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.262+0000 7fbe537fe640 10 monclient(hunting): my global_id is 4340 2024-03-25T18:10:30.263 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.262+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe3c004eb0 con 0x7fbe5c14be90 2024-03-25T18:10:30.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.266+0000 7fbe537fe640 10 client.?.objecter ms_handle_connect 0x7fbe5c0b4d60 2024-03-25T18:10:30.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.266+0000 7fbe537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:30.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.268+0000 7fbe537fe640 10 client.?.objecter ms_handle_connect 0x7fbe5c0b1730 2024-03-25T18:10:30.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.268+0000 7fbe537fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:30.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2193721098 0 0) 0x7fbe5c141a00 con 0x7fbe5c0b4d60 2024-03-25T18:10:30.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:30.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:30.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:30.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe5c1d7a90 con 0x7fbe5c0b4d60 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3305628906 0 0) 0x7fbe5c141790 con 0x7fbe5c0b1730 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): _negotiate 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe5c141a00 con 0x7fbe5c0b1730 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 820494174 0 0) 0x7fbe3c004eb0 con 0x7fbe5c14be90 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe5c141790 con 0x7fbe5c14be90 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbe54001c90 con 0x7fbe5c14be90 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): dump: 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:30.280 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:30.281 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:30.281 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:30.281 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:30.281 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:30.281 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:30.281 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:30.281 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:30.281 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:30.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.278+0000 7fbe537fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:30.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2951486373 0 0) 0x7fbe5c141a00 con 0x7fbe5c0b1730 2024-03-25T18:10:30.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe3c004eb0 con 0x7fbe5c0b1730 2024-03-25T18:10:30.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbe4c001930 con 0x7fbe5c0b1730 2024-03-25T18:10:30.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:30.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:30.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 10 monclient(hunting): dump: 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2797964739 0 0) 0x7fbe5c141790 con 0x7fbe5c14be90 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 >> v1:172.21.15.117:6789/0 conn(0x7fbe5c0b4d60 legacy=0x7fbe5c142a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:30.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 >> v1:172.21.15.37:6789/0 conn(0x7fbe5c0b1730 legacy=0x7fbe5c13b840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 10 monclient: _finish_hunting 0 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 1 monclient: found mon.c 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe5c143140 con 0x7fbe5c14be90 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 10 monclient: _finish_auth 0 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 10 monclient: _check_auth_tickets 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe6100f640 5 monclient: authenticate success, global_id 4340 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe6100f640 10 monclient: _renew_subs 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe6100f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.287+0000 7fbe6100f640 1 -- 172.21.15.37:0/3945085449 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe5c1d86c0 con 0x7fbe5c14be90 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.288+0000 7fbe6100f640 10 client.4340.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:30.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.288+0000 7fbe6100f640 10 monclient: _renew_subs 2024-03-25T18:10:30.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.288+0000 7fbe6100f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:10:30.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.288+0000 7fbe6100f640 1 -- 172.21.15.37:0/3945085449 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe5c1422e0 con 0x7fbe5c14be90 2024-03-25T18:10:30.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.288+0000 7fbe6100f640 1 librados: init done 2024-03-25T18:10:30.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.289+0000 7fbe6100f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:30.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.289+0000 7fbe6100f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:30.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.289+0000 7fbe6100f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:10:30.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.289+0000 7fbe6100f640 1 -- 172.21.15.37:0/3945085449 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe5c0627b0 con 0x7fbe5c14be90 2024-03-25T18:10:30.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.294+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe54003990 con 0x7fbe5c14be90 2024-03-25T18:10:30.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.294+0000 7fbe537fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:30.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.294+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbe54003ce0 con 0x7fbe5c14be90 2024-03-25T18:10:30.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.294+0000 7fbe537fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:30.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.294+0000 7fbe537fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:30.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.294+0000 7fbe537fe640 10 monclient: dump: 2024-03-25T18:10:30.295 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:30.295 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:30.295 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:30.296 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:30.296 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:30.296 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:30.296 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:30.296 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:30.296 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:30.296 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:30.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.294+0000 7fbe59d9b640 4 set_mon_vals no callback set 2024-03-25T18:10:30.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.607+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fbe54003e40 con 0x7fbe5c14be90 2024-03-25T18:10:30.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.608+0000 7fbe5ad9d640 10 monclient: get_auth_request con 0x7fbe3c06df70 auth_method 0 2024-03-25T18:10:30.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.608+0000 7fbe537fe640 10 client.4340.objecter ms_handle_connect 0x7fbe3c06df70 2024-03-25T18:10:30.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.614+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fbe54054980 con 0x7fbe5c14be90 2024-03-25T18:10:30.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.614+0000 7fbe537fe640 10 client.4340.objecter ms_dispatch 0x7fbe5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:30.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.614+0000 7fbe537fe640 3 client.4340.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:30.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.614+0000 7fbe537fe640 3 client.4340.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:30.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.614+0000 7fbe537fe640 20 client.4340.objecter dump_active .. 0 homeless 2024-03-25T18:10:30.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.614+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe54050490 con 0x7fbe5c14be90 2024-03-25T18:10:30.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.614+0000 7fbe537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:30.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.614+0000 7fbe537fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:30.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.840+0000 7fbe6100f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:30.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.840+0000 7fbe6100f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:30.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.840+0000 7fbe6100f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:10:30.842 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.840+0000 7fbe6100f640 1 -- 172.21.15.37:0/3945085449 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe5c14bbc0 con 0x7fbe5c14be90 2024-03-25T18:10:30.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.876+0000 7fbe537fe640 1 -- 172.21.15.37:0/3945085449 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fbe540523c0 con 0x7fbe5c14be90 2024-03-25T18:10:30.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.877+0000 7fbe537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:30.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.877+0000 7fbe537fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:30.878 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:10:30.878 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:30.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 10 librados: watch_flush enter 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 10 librados: watch_flush exit 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 20 client.4340.objecter shutdown clearing up homeless session... 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 10 client.4340.objecter successfully canceled tick 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 1 -- 172.21.15.37:0/3945085449 >> v1:172.21.15.117:6800/3703993921 conn(0x7fbe3c06df70 legacy=0x7fbe3c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 10 monclient: shutdown 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 1 -- 172.21.15.37:0/3945085449 >> v1:172.21.15.37:6790/0 conn(0x7fbe5c14be90 legacy=0x7fbe5c1d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 1 -- 172.21.15.37:0/3945085449 shutdown_connections 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 1 -- 172.21.15.37:0/3945085449 >> 172.21.15.37:0/3945085449 conn(0x7fbe5c0617a0 msgr2=0x7fbe5c0b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 1 -- 172.21.15.37:0/3945085449 shutdown_connections 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.879+0000 7fbe6100f640 1 -- 172.21.15.37:0/3945085449 wait complete. 2024-03-25T18:10:30.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:30.880+0000 7fbe6100f640 1 librados: shutdown 2024-03-25T18:10:30.893 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:37.699 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.819+0000 7f7315e9d640 10 monclient: get_monmap_and_config 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.819+0000 7f7315e9d640 10 monclient: build_initial_monmap 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.819+0000 7f7315e9d640 10 monclient: monmap: 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:10:37.820829+0000 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:10:37.820829+0000 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:10:37.821 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:37.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.820+0000 7f7315e9d640 1 Processor -- start 2024-03-25T18:10:37.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.820+0000 7f7315e9d640 1 -- start start 2024-03-25T18:10:37.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.820+0000 7f7315e9d640 10 monclient: init 2024-03-25T18:10:37.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.821+0000 7f7315e9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:37.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.821+0000 7f7315e9d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:10:37.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.821+0000 7f7315e9d640 10 monclient(hunting): picked mon.noname-b con 0x7f731006c380 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:37.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.821+0000 7f7315e9d640 10 monclient(hunting): picked mon.noname-c con 0x7f73100b12d0 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:37.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.821+0000 7f7315e9d640 10 monclient(hunting): picked mon.noname-a con 0x7f73100aedf0 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:37.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.821+0000 7f7315e9d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73100b16f0 con 0x7f73100aedf0 2024-03-25T18:10:37.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.821+0000 7f7315e9d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73100adfe0 con 0x7f73100b12d0 2024-03-25T18:10:37.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.821+0000 7f7315e9d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73100add00 con 0x7f731006c380 2024-03-25T18:10:37.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.821+0000 7f7315e9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:37.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.821+0000 7f7315e9d640 10 monclient(hunting): authenticate will time out at 6993.074025s 2024-03-25T18:10:37.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.825+0000 7f730effd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f73100b12d0 0x7f73100ae6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:49018/0 (socket says 172.21.15.37:49018) 2024-03-25T18:10:37.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.825+0000 7f730effd640 1 -- 172.21.15.37:0/3354200379 learned_addr learned my addr 172.21.15.37:0/3354200379 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:37.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.833+0000 7f730e7fc640 1 -- 172.21.15.37:0/3354200379 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 628901962 0 0) 0x7f73100b16f0 con 0x7f73100aedf0 2024-03-25T18:10:37.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.833+0000 7f730e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:37.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.833+0000 7f730e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:37.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.833+0000 7f730e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:37.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.833+0000 7f730e7fc640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:10:37.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.833+0000 7f730e7fc640 1 -- 172.21.15.37:0/3354200379 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72f8003520 con 0x7f73100aedf0 2024-03-25T18:10:37.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.838+0000 7f730e7fc640 1 -- 172.21.15.37:0/3354200379 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583136548 0 0) 0x7f73100adfe0 con 0x7f73100b12d0 2024-03-25T18:10:37.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.838+0000 7f730e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:10:37.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.838+0000 7f730e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:37.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.838+0000 7f730e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:37.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.838+0000 7f730e7fc640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:10:37.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.838+0000 7f730e7fc640 1 -- 172.21.15.37:0/3354200379 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73100b16f0 con 0x7f73100b12d0 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 1 -- 172.21.15.37:0/3354200379 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 212927808 0 0) 0x7f72f8003520 con 0x7f73100aedf0 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 1 -- 172.21.15.37:0/3354200379 >> v1:172.21.15.37:6790/0 conn(0x7f731006c380 legacy=0x7f73100b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 1 -- 172.21.15.37:0/3354200379 >> v1:172.21.15.117:6789/0 conn(0x7f73100b12d0 legacy=0x7f73100ae6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 1 -- 172.21.15.37:0/3354200379 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73100ae2c0 con 0x7f73100aedf0 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 1 -- 172.21.15.37:0/3354200379 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7308001a10 con 0x7f73100aedf0 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:37.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f730e7fc640 10 monclient: dump: 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f7315e9d640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:10:37.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.842+0000 7f7315e9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.867+0000 7f730e7fc640 1 -- 172.21.15.37:0/3354200379 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7308001b70 con 0x7f73100aedf0 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.867+0000 7f730e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.867+0000 7f730e7fc640 1 -- 172.21.15.37:0/3354200379 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7308001f00 con 0x7f73100aedf0 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.867+0000 7f7315e9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.867+0000 7f7315e9d640 4 set_mon_vals no callback set 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.867+0000 7f730e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.867+0000 7f730e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.867+0000 7f730e7fc640 10 monclient: dump: 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:37.869 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:37.870 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:37.870 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:37.870 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:37.870 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:37.870 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:37.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.868+0000 7f7315e9d640 10 monclient: shutdown 2024-03-25T18:10:37.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.868+0000 7f7315e9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:37.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.868+0000 7f7315e9d640 1 -- 172.21.15.37:0/3354200379 >> v1:172.21.15.37:6789/0 conn(0x7f73100aedf0 legacy=0x7f731014e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:37.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.868+0000 7f7315e9d640 1 -- 172.21.15.37:0/3354200379 shutdown_connections 2024-03-25T18:10:37.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.868+0000 7f7315e9d640 1 -- 172.21.15.37:0/3354200379 >> 172.21.15.37:0/3354200379 conn(0x7f73100b5d10 msgr2=0x7f73100b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.868+0000 7f7315e9d640 1 -- 172.21.15.37:0/3354200379 shutdown_connections 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.868+0000 7f7315e9d640 1 -- 172.21.15.37:0/3354200379 wait complete. 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.869+0000 7f7315e9d640 10 monclient: build_initial_monmap 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.869+0000 7f7315e9d640 10 monclient: monmap: 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:37.872 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.870+0000 7f7315e9d640 1 librados: starting msgr at 2024-03-25T18:10:37.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.870+0000 7f7315e9d640 1 librados: starting objecter 2024-03-25T18:10:37.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.870+0000 7f7315e9d640 1 Processor -- start 2024-03-25T18:10:37.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.870+0000 7f7315e9d640 1 -- start start 2024-03-25T18:10:37.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.870+0000 7f7315e9d640 1 librados: setting wanted keys 2024-03-25T18:10:37.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.870+0000 7f7315e9d640 1 librados: calling monclient init 2024-03-25T18:10:37.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.870+0000 7f7315e9d640 10 monclient: init 2024-03-25T18:10:37.874 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.871+0000 7f7315e9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:37.874 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.871+0000 7f7315e9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:10:37.874 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.871+0000 7f7315e9d640 10 monclient(hunting): picked mon.noname-a con 0x7f731006c380 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:37.874 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.871+0000 7f7315e9d640 10 monclient(hunting): picked mon.noname-b con 0x7f73100aedf0 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:37.874 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.871+0000 7f7315e9d640 10 monclient(hunting): picked mon.noname-c con 0x7f73100b12d0 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:37.874 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.871+0000 7f7315e9d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f731006c8e0 con 0x7f731006c380 2024-03-25T18:10:37.874 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.871+0000 7f7315e9d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73101a32c0 con 0x7f73100aedf0 2024-03-25T18:10:37.874 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.871+0000 7f7315e9d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73101a4240 con 0x7f73100b12d0 2024-03-25T18:10:37.874 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.871+0000 7f7315e9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:37.875 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.871+0000 7f7315e9d640 10 monclient(hunting): authenticate will time out at 6993.123965s 2024-03-25T18:10:37.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.877+0000 7f730ffff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f73100b12d0 0x7f73101dab30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:60900/0 (socket says 172.21.15.37:60900) 2024-03-25T18:10:37.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.877+0000 7f730ffff640 1 -- 172.21.15.37:0/3799626614 learned_addr learned my addr 172.21.15.37:0/3799626614 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:37.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.879+0000 7f72f7fff640 10 client.?.objecter ms_handle_connect 0x7f73100b12d0 2024-03-25T18:10:37.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.879+0000 7f72f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:37.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.884+0000 7f72f7fff640 10 client.?.objecter ms_handle_connect 0x7f731006c380 2024-03-25T18:10:37.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.884+0000 7f72f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:37.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.894+0000 7f72f7fff640 10 client.?.objecter ms_handle_connect 0x7f73100aedf0 2024-03-25T18:10:37.894 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.894+0000 7f72f7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:37.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.899+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 210692595 0 0) 0x7f731006c8e0 con 0x7f731006c380 2024-03-25T18:10:37.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.899+0000 7f72f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:37.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.899+0000 7f72f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:37.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.899+0000 7f72f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:37.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.899+0000 7f72f7fff640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:10:37.900 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.899+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72fc004eb0 con 0x7f731006c380 2024-03-25T18:10:37.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.912+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2114291676 0 0) 0x7f73101a4240 con 0x7f73100b12d0 2024-03-25T18:10:37.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.912+0000 7f72f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:37.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.912+0000 7f72f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:37.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.912+0000 7f72f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:37.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.912+0000 7f72f7fff640 10 monclient(hunting): my global_id is 4346 2024-03-25T18:10:37.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.912+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f731006c8e0 con 0x7f73100b12d0 2024-03-25T18:10:37.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.913+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3348066742 0 0) 0x7f73101a32c0 con 0x7f73100aedf0 2024-03-25T18:10:37.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.913+0000 7f72f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:37.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.913+0000 7f72f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:37.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.913+0000 7f72f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:37.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.913+0000 7f72f7fff640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:10:37.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.913+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73101a4240 con 0x7f73100aedf0 2024-03-25T18:10:37.918 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.917+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2120276234 0 0) 0x7f72fc004eb0 con 0x7f731006c380 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.917+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f73101a32c0 con 0x7f731006c380 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.917+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7300001930 con 0x7f731006c380 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.917+0000 7f72f7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.917+0000 7f72f7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.917+0000 7f72f7fff640 10 monclient(hunting): dump: 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:37.919 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:37.920 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:37.920 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.918+0000 7f72f7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:37.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3695560875 0 0) 0x7f73101a32c0 con 0x7f731006c380 2024-03-25T18:10:37.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 >> v1:172.21.15.37:6790/0 conn(0x7f73100b12d0 legacy=0x7f73101dab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:37.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 >> v1:172.21.15.117:6789/0 conn(0x7f73100aedf0 legacy=0x7f73101c3ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:37.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f72f7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:37.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f72f7fff640 1 monclient: found mon.a 2024-03-25T18:10:37.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f72f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:37.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f72f7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:10:37.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73101c4f90 con 0x7f731006c380 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f72f7fff640 10 monclient: _finish_auth 0 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f72f7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f72f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f7315e9d640 5 monclient: authenticate success, global_id 4350 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f7315e9d640 10 monclient: _renew_subs 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f7315e9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f7315e9d640 1 -- 172.21.15.37:0/3799626614 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73101c45b0 con 0x7f731006c380 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f7315e9d640 10 client.4350.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f7315e9d640 10 monclient: _renew_subs 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f7315e9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f7315e9d640 1 -- 172.21.15.37:0/3799626614 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f73101c4900 con 0x7f731006c380 2024-03-25T18:10:37.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.928+0000 7f7315e9d640 1 librados: init done 2024-03-25T18:10:37.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.929+0000 7f7315e9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:37.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.929+0000 7f7315e9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:37.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.929+0000 7f7315e9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:10:37.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:37.929+0000 7f7315e9d640 1 -- 172.21.15.37:0/3799626614 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73100627b0 con 0x7f731006c380 2024-03-25T18:10:38.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.233+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f73000035f0 con 0x7f731006c380 2024-03-25T18:10:38.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.233+0000 7f72f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.233+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f73000039c0 con 0x7f731006c380 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.233+0000 7f72f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.233+0000 7f72f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.233+0000 7f72f7fff640 10 monclient: dump: 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:38.235 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:38.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.233+0000 7f730dffb640 4 set_mon_vals no callback set 2024-03-25T18:10:38.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.234+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f7300003be0 con 0x7f731006c380 2024-03-25T18:10:38.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.235+0000 7f730effd640 10 monclient: get_auth_request con 0x7f72fc06de20 auth_method 0 2024-03-25T18:10:38.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.235+0000 7f72f7fff640 10 client.4350.objecter ms_handle_connect 0x7f72fc06de20 2024-03-25T18:10:38.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.241+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f7300053630 con 0x7f731006c380 2024-03-25T18:10:38.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.241+0000 7f72f7fff640 10 client.4350.objecter ms_dispatch 0x7f7310000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:38.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.241+0000 7f72f7fff640 3 client.4350.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:38.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.241+0000 7f72f7fff640 3 client.4350.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:38.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.241+0000 7f72f7fff640 20 client.4350.objecter dump_active .. 0 homeless 2024-03-25T18:10:38.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.241+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f730004f140 con 0x7f731006c380 2024-03-25T18:10:38.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.241+0000 7f72f7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:38.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.241+0000 7f72f7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:38.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.474+0000 7f7315e9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:38.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.474+0000 7f7315e9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:38.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.474+0000 7f7315e9d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:10:38.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.474+0000 7f7315e9d640 1 -- 172.21.15.37:0/3799626614 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7310069f40 con 0x7f731006c380 2024-03-25T18:10:38.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.516+0000 7f72f7fff640 1 -- 172.21.15.37:0/3799626614 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f7300051070 con 0x7f731006c380 2024-03-25T18:10:38.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.516+0000 7f72f7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:38.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.516+0000 7f72f7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:38.518 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:10:38.518 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:38.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 10 librados: watch_flush enter 2024-03-25T18:10:38.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 10 librados: watch_flush exit 2024-03-25T18:10:38.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 20 client.4350.objecter shutdown clearing up homeless session... 2024-03-25T18:10:38.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 10 client.4350.objecter successfully canceled tick 2024-03-25T18:10:38.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 1 -- 172.21.15.37:0/3799626614 >> v1:172.21.15.117:6800/3703993921 conn(0x7f72fc06de20 legacy=0x7f72fc0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:38.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 10 monclient: shutdown 2024-03-25T18:10:38.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:38.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 1 -- 172.21.15.37:0/3799626614 >> v1:172.21.15.37:6789/0 conn(0x7f731006c380 legacy=0x7f73101a3b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:38.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 1 -- 172.21.15.37:0/3799626614 shutdown_connections 2024-03-25T18:10:38.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 1 -- 172.21.15.37:0/3799626614 >> 172.21.15.37:0/3799626614 conn(0x7f73100b5d10 msgr2=0x7f73101cd4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:38.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 1 -- 172.21.15.37:0/3799626614 shutdown_connections 2024-03-25T18:10:38.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 1 -- 172.21.15.37:0/3799626614 wait complete. 2024-03-25T18:10:38.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:38.519+0000 7f7315e9d640 1 librados: shutdown 2024-03-25T18:10:38.531 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:44.521 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:03:19.867 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:44.535 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:03:19.880 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:44.568 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:03:19.913 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:44.577 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:03:19.923 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:44.579 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:03:19.893 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:44.583 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:03:19.896 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:44.587 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:03:19.933 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:44.612 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:03:19.958 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:44.618 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:03:19.931 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:10:45.337 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:45.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.454+0000 7f114fa57640 10 monclient: get_monmap_and_config 2024-03-25T18:10:45.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.454+0000 7f114fa57640 10 monclient: build_initial_monmap 2024-03-25T18:10:45.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.454+0000 7f114fa57640 10 monclient: monmap: 2024-03-25T18:10:45.455 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:45.455 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:45.455 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:10:45.455202+0000 2024-03-25T18:10:45.455 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:10:45.455202+0000 2024-03-25T18:10:45.455 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:45.456 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:45.456 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:45.456 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:10:45.456 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:10:45.456 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:45.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.455+0000 7f114fa57640 1 Processor -- start 2024-03-25T18:10:45.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.455+0000 7f114fa57640 1 -- start start 2024-03-25T18:10:45.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.455+0000 7f114fa57640 10 monclient: init 2024-03-25T18:10:45.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.456+0000 7f114fa57640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:45.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.456+0000 7f114fa57640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:45.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.456+0000 7f114fa57640 10 monclient(hunting): picked mon.noname-a con 0x7f11480a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:45.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.456+0000 7f114fa57640 10 monclient(hunting): picked mon.noname-b con 0x7f114813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:45.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.456+0000 7f114fa57640 10 monclient(hunting): picked mon.noname-c con 0x7f11481430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:45.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.456+0000 7f114fa57640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1148142520 con 0x7f11480a98f0 2024-03-25T18:10:45.457 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.456+0000 7f114fa57640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11480b28a0 con 0x7f11481430c0 2024-03-25T18:10:45.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.456+0000 7f114fa57640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11480b2a30 con 0x7f114813f710 2024-03-25T18:10:45.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.456+0000 7f114fa57640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:45.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.456+0000 7f114fa57640 10 monclient(hunting): authenticate will time out at 7000.708729s 2024-03-25T18:10:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.468+0000 7f114d7f3640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f11480a98f0 0x7f11480a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49218/0 (socket says 172.21.15.37:49218) 2024-03-25T18:10:45.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.468+0000 7f114d7f3640 1 -- 172.21.15.37:0/2697757458 learned_addr learned my addr 172.21.15.37:0/2697757458 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:45.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.480+0000 7f113ffff640 1 -- 172.21.15.37:0/2697757458 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3611721718 0 0) 0x7f1148142520 con 0x7f11480a98f0 2024-03-25T18:10:45.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.480+0000 7f113ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:45.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.480+0000 7f113ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:45.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.480+0000 7f113ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:45.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.480+0000 7f113ffff640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:10:45.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.480+0000 7f113ffff640 1 -- 172.21.15.37:0/2697757458 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1134003520 con 0x7f11480a98f0 2024-03-25T18:10:45.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.489+0000 7f113ffff640 1 -- 172.21.15.37:0/2697757458 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4223657004 0 0) 0x7f11480b28a0 con 0x7f11481430c0 2024-03-25T18:10:45.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.489+0000 7f113ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:10:45.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.489+0000 7f113ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:45.489 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.489+0000 7f113ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:45.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.489+0000 7f113ffff640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:10:45.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.489+0000 7f113ffff640 1 -- 172.21.15.37:0/2697757458 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1148142520 con 0x7f11481430c0 2024-03-25T18:10:45.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 1 -- 172.21.15.37:0/2697757458 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2821758195 0 0) 0x7f1134003520 con 0x7f11480a98f0 2024-03-25T18:10:45.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 1 -- 172.21.15.37:0/2697757458 >> v1:172.21.15.37:6790/0 conn(0x7f114813f710 legacy=0x7f11481429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:45.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 1 -- 172.21.15.37:0/2697757458 >> v1:172.21.15.117:6789/0 conn(0x7f11481430c0 legacy=0x7f114814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:45.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:10:45.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 1 monclient: found mon.noname-a 2024-03-25T18:10:45.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:45.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:10:45.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 1 -- 172.21.15.37:0/2697757458 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11480b44c0 con 0x7f11480a98f0 2024-03-25T18:10:45.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 10 monclient: _finish_auth 0 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 1 -- 172.21.15.37:0/2697757458 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1144001690 con 0x7f11480a98f0 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f113ffff640 10 monclient: dump: 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:45.498 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:45.499 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:45.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f114fa57640 5 monclient: authenticate success, global_id 4353 2024-03-25T18:10:45.499 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.496+0000 7f114fa57640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:45.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.508+0000 7f113ffff640 1 -- 172.21.15.37:0/2697757458 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11440017f0 con 0x7f11480a98f0 2024-03-25T18:10:45.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.508+0000 7f113ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:45.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.508+0000 7f113ffff640 1 -- 172.21.15.37:0/2697757458 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1144001c00 con 0x7f11480a98f0 2024-03-25T18:10:45.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.508+0000 7f113ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:45.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.508+0000 7f113ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:45.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.508+0000 7f113ffff640 10 monclient: dump: 2024-03-25T18:10:45.509 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:45.510 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:45.510 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.510 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.510 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:45.510 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:45.510 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:45.510 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:45.510 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:45.510 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:45.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.508+0000 7f114fa57640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:45.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.508+0000 7f114fa57640 4 set_mon_vals no callback set 2024-03-25T18:10:45.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.509+0000 7f114fa57640 10 monclient: shutdown 2024-03-25T18:10:45.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.509+0000 7f114fa57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:45.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.509+0000 7f114fa57640 1 -- 172.21.15.37:0/2697757458 >> v1:172.21.15.37:6789/0 conn(0x7f11480a98f0 legacy=0x7f11480a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:45.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.509+0000 7f114fa57640 1 -- 172.21.15.37:0/2697757458 shutdown_connections 2024-03-25T18:10:45.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.509+0000 7f114fa57640 1 -- 172.21.15.37:0/2697757458 >> 172.21.15.37:0/2697757458 conn(0x7f11480b7ea0 msgr2=0x7f11480b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:45.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.509+0000 7f114fa57640 1 -- 172.21.15.37:0/2697757458 shutdown_connections 2024-03-25T18:10:45.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.509+0000 7f114fa57640 1 -- 172.21.15.37:0/2697757458 wait complete. 2024-03-25T18:10:45.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.510+0000 7f114fa57640 10 monclient: build_initial_monmap 2024-03-25T18:10:45.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.510+0000 7f114fa57640 10 monclient: monmap: 2024-03-25T18:10:45.512 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:45.512 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.511+0000 7f114fa57640 1 librados: starting msgr at 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.511+0000 7f114fa57640 1 librados: starting objecter 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.511+0000 7f114fa57640 1 Processor -- start 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.511+0000 7f114fa57640 1 -- start start 2024-03-25T18:10:45.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.511+0000 7f114fa57640 1 librados: setting wanted keys 2024-03-25T18:10:45.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.511+0000 7f114fa57640 1 librados: calling monclient init 2024-03-25T18:10:45.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.511+0000 7f114fa57640 10 monclient: init 2024-03-25T18:10:45.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.512+0000 7f114fa57640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:45.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.512+0000 7f114fa57640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:45.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.512+0000 7f114fa57640 10 monclient(hunting): picked mon.noname-b con 0x7f11480a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:45.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.512+0000 7f114fa57640 10 monclient(hunting): picked mon.noname-a con 0x7f114813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:45.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.512+0000 7f114fa57640 10 monclient(hunting): picked mon.noname-c con 0x7f11481430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:45.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.512+0000 7f114fa57640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11481d7910 con 0x7f114813f710 2024-03-25T18:10:45.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.512+0000 7f114fa57640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11480abfe0 con 0x7f11480a98f0 2024-03-25T18:10:45.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.512+0000 7f114fa57640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11481d84b0 con 0x7f11481430c0 2024-03-25T18:10:45.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.512+0000 7f114fa57640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:45.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.512+0000 7f114fa57640 10 monclient(hunting): authenticate will time out at 7000.764969s 2024-03-25T18:10:45.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.514+0000 7f114cff2640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f114813f710 0x7f11481385f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:49248/0 (socket says 172.21.15.37:49248) 2024-03-25T18:10:45.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.514+0000 7f114cff2640 1 -- 172.21.15.37:0/2076416041 learned_addr learned my addr 172.21.15.37:0/2076416041 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:45.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.515+0000 7f113dffb640 10 client.?.objecter ms_handle_connect 0x7f11480a98f0 2024-03-25T18:10:45.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.516+0000 7f113dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:45.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.516+0000 7f113dffb640 10 client.?.objecter ms_handle_connect 0x7f114813f710 2024-03-25T18:10:45.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.516+0000 7f113dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:45.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.524+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3352243388 0 0) 0x7f11480abfe0 con 0x7f11480a98f0 2024-03-25T18:10:45.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.524+0000 7f113dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:45.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.524+0000 7f113dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:45.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.524+0000 7f113dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:45.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.524+0000 7f113dffb640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:10:45.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.524+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f112c004eb0 con 0x7f11480a98f0 2024-03-25T18:10:45.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.526+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3871224026 0 0) 0x7f11481d7910 con 0x7f114813f710 2024-03-25T18:10:45.527 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.526+0000 7f113dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:45.528 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.526+0000 7f113dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:45.528 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.526+0000 7f113dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:45.528 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.526+0000 7f113dffb640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:10:45.528 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.526+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11480abfe0 con 0x7f114813f710 2024-03-25T18:10:45.528 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.527+0000 7f113dffb640 10 client.?.objecter ms_handle_connect 0x7f11481430c0 2024-03-25T18:10:45.528 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.527+0000 7f113dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.537+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 66333028 0 0) 0x7f112c004eb0 con 0x7f11480a98f0 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.537+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11481d7910 con 0x7f11480a98f0 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.537+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1144001bc0 con 0x7f11480a98f0 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.537+0000 7f113dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.537+0000 7f113dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.537+0000 7f113dffb640 10 monclient(hunting): dump: 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:45.539 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:45.540 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:45.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.537+0000 7f113dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:45.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.538+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3770490703 0 0) 0x7f11481d84b0 con 0x7f11481430c0 2024-03-25T18:10:45.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.538+0000 7f113dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:10:45.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.538+0000 7f113dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:45.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.538+0000 7f113dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:45.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.538+0000 7f113dffb640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:10:45.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.538+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f112c004eb0 con 0x7f11481430c0 2024-03-25T18:10:45.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.545+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2163193173 0 0) 0x7f11480abfe0 con 0x7f114813f710 2024-03-25T18:10:45.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.545+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11481d84b0 con 0x7f114813f710 2024-03-25T18:10:45.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.545+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1138001990 con 0x7f114813f710 2024-03-25T18:10:45.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.545+0000 7f113dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.545+0000 7f113dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.545+0000 7f113dffb640 10 monclient(hunting): dump: 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:45.547 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:45.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.551+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1651123674 0 0) 0x7f112c004eb0 con 0x7f11481430c0 2024-03-25T18:10:45.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.551+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11480abfe0 con 0x7f11481430c0 2024-03-25T18:10:45.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.551+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1140001bd0 con 0x7f11481430c0 2024-03-25T18:10:45.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.551+0000 7f113dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:45.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.551+0000 7f113dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:45.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.551+0000 7f113dffb640 10 monclient(hunting): dump: 2024-03-25T18:10:45.552 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:45.553 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:45.553 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.553 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.553 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:45.553 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:45.553 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:45.553 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:45.553 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:45.553 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:45.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3530595212 0 0) 0x7f11481d7910 con 0x7f11480a98f0 2024-03-25T18:10:45.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 >> v1:172.21.15.37:6790/0 conn(0x7f11481430c0 legacy=0x7f1148138d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:45.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 >> v1:172.21.15.37:6789/0 conn(0x7f114813f710 legacy=0x7f11481385f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:45.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f113dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:10:45.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f113dffb640 1 monclient: found mon.b 2024-03-25T18:10:45.567 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f113dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f113dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11481d51f0 con 0x7f11480a98f0 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f113dffb640 10 monclient: _finish_auth 0 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f113dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f113dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f114fa57640 5 monclient: authenticate success, global_id 4357 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f114fa57640 10 monclient: _renew_subs 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f114fa57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f114fa57640 1 -- 172.21.15.37:0/2076416041 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11481e2ed0 con 0x7f11480a98f0 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f114fa57640 10 client.4357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f114fa57640 10 monclient: _renew_subs 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f114fa57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f114fa57640 1 -- 172.21.15.37:0/2076416041 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11481e3240 con 0x7f11480a98f0 2024-03-25T18:10:45.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.566+0000 7f114fa57640 1 librados: init done 2024-03-25T18:10:45.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.568+0000 7f114fa57640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:45.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.568+0000 7f114fa57640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:45.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.568+0000 7f114fa57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:45.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.568+0000 7f114fa57640 1 -- 172.21.15.37:0/2076416041 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11480627b0 con 0x7f11480a98f0 2024-03-25T18:10:45.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.832+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1144004a90 con 0x7f11480a98f0 2024-03-25T18:10:45.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.832+0000 7f113dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:45.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.832+0000 7f113f7fe640 4 set_mon_vals no callback set 2024-03-25T18:10:45.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.846+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1144001250 con 0x7f11480a98f0 2024-03-25T18:10:45.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.846+0000 7f113dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:45.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.846+0000 7f113dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:45.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.846+0000 7f113dffb640 10 monclient: dump: 2024-03-25T18:10:45.847 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:45.847 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:45.847 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.847 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:45.847 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:45.848 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:45.848 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:45.848 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:45.848 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:45.848 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:45.848 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.847+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f1144004e20 con 0x7f11480a98f0 2024-03-25T18:10:45.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.847+0000 7f114cff2640 10 monclient: get_auth_request con 0x7f112c06e110 auth_method 0 2024-03-25T18:10:45.849 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.848+0000 7f113dffb640 10 client.4357.objecter ms_handle_connect 0x7f112c06e110 2024-03-25T18:10:45.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.851+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f11440548f0 con 0x7f11480a98f0 2024-03-25T18:10:45.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.851+0000 7f113dffb640 10 client.4357.objecter ms_dispatch 0x7f1148000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:45.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.851+0000 7f113dffb640 3 client.4357.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:45.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.851+0000 7f113dffb640 3 client.4357.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:45.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.851+0000 7f113dffb640 20 client.4357.objecter dump_active .. 0 homeless 2024-03-25T18:10:45.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.852+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11440b18b0 con 0x7f11480a98f0 2024-03-25T18:10:45.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.852+0000 7f113dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:45.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:45.852+0000 7f113dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:46.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.092+0000 7f114fa57640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:46.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.092+0000 7f114fa57640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:46.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.092+0000 7f114fa57640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:46.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.092+0000 7f114fa57640 1 -- 172.21.15.37:0/2076416041 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f114813df50 con 0x7f11480a98f0 2024-03-25T18:10:46.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.130+0000 7f113dffb640 1 -- 172.21.15.37:0/2076416041 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f1144050400 con 0x7f11480a98f0 2024-03-25T18:10:46.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.130+0000 7f113dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:46.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.130+0000 7f113dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:46.132 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:10:46.132 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 10 librados: watch_flush enter 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 10 librados: watch_flush exit 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 20 client.4357.objecter shutdown clearing up homeless session... 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 10 client.4357.objecter successfully canceled tick 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 1 -- 172.21.15.37:0/2076416041 >> v1:172.21.15.117:6800/3703993921 conn(0x7f112c06e110 legacy=0x7f112c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 10 monclient: shutdown 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 1 -- 172.21.15.37:0/2076416041 >> v1:172.21.15.117:6789/0 conn(0x7f11480a98f0 legacy=0x7f11481d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 1 -- 172.21.15.37:0/2076416041 shutdown_connections 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 1 -- 172.21.15.37:0/2076416041 >> 172.21.15.37:0/2076416041 conn(0x7f11480b7ea0 msgr2=0x7f1148074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 1 -- 172.21.15.37:0/2076416041 shutdown_connections 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 1 -- 172.21.15.37:0/2076416041 wait complete. 2024-03-25T18:10:46.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:46.133+0000 7f114fa57640 1 librados: shutdown 2024-03-25T18:10:46.145 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:10:52.952 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.070+0000 7f8af4ff6640 10 monclient: get_monmap_and_config 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.070+0000 7f8af4ff6640 10 monclient: build_initial_monmap 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.070+0000 7f8af4ff6640 10 monclient: monmap: 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:10:53.072208+0000 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:10:53.072208+0000 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:10:53.072 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:10:53.073 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.071+0000 7f8af4ff6640 1 Processor -- start 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.071+0000 7f8af4ff6640 1 -- start start 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.071+0000 7f8af4ff6640 10 monclient: init 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.071+0000 7f8af4ff6640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.071+0000 7f8af4ff6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.071+0000 7f8af4ff6640 10 monclient(hunting): picked mon.noname-a con 0x7f8af0061830 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.071+0000 7f8af4ff6640 10 monclient(hunting): picked mon.noname-b con 0x7f8af00ae840 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.071+0000 7f8af4ff6640 10 monclient(hunting): picked mon.noname-c con 0x7f8af013fbd0 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.071+0000 7f8af4ff6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af00ae1c0 con 0x7f8af0061830 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.071+0000 7f8af4ff6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af006f7d0 con 0x7f8af013fbd0 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.072+0000 7f8af4ff6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af006fa40 con 0x7f8af00ae840 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.072+0000 7f8af4ff6640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:53.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.072+0000 7f8af4ff6640 10 monclient(hunting): authenticate will time out at 7008.325304s 2024-03-25T18:10:53.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.081+0000 7f8aedd9b640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8af00ae840 0x7f8af0145810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:58862/0 (socket says 172.21.15.37:58862) 2024-03-25T18:10:53.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.081+0000 7f8aedd9b640 1 -- 172.21.15.37:0/17410597 learned_addr learned my addr 172.21.15.37:0/17410597 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.095+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055716688 0 0) 0x7f8af006fa40 con 0x7f8af00ae840 2024-03-25T18:10:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.096+0000 7f8aed59a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:53.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.096+0000 7f8aed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.096+0000 7f8aed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.096+0000 7f8aed59a640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:10:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.096+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad8003520 con 0x7f8af00ae840 2024-03-25T18:10:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.096+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390610188 0 0) 0x7f8af00ae1c0 con 0x7f8af0061830 2024-03-25T18:10:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.096+0000 7f8aed59a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:53.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.096+0000 7f8aed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:53.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.096+0000 7f8aed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:53.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.096+0000 7f8aed59a640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:10:53.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.096+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af006fa40 con 0x7f8af0061830 2024-03-25T18:10:53.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.098+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788427008 0 0) 0x7f8af006f7d0 con 0x7f8af013fbd0 2024-03-25T18:10:53.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.098+0000 7f8aed59a640 10 monclient(hunting): _negotiate 2024-03-25T18:10:53.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.098+0000 7f8aed59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:53.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.098+0000 7f8aed59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:53.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.098+0000 7f8aed59a640 10 monclient(hunting): my global_id is 4360 2024-03-25T18:10:53.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.098+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af00ae1c0 con 0x7f8af013fbd0 2024-03-25T18:10:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4249520588 0 0) 0x7f8ad8003520 con 0x7f8af00ae840 2024-03-25T18:10:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 >> v1:172.21.15.117:6789/0 conn(0x7f8af013fbd0 legacy=0x7f8af0145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 >> v1:172.21.15.37:6789/0 conn(0x7f8af0061830 legacy=0x7f8af013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:53.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8aed59a640 10 monclient: _finish_hunting 0 2024-03-25T18:10:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8aed59a640 1 monclient: found mon.noname-b 2024-03-25T18:10:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8aed59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8aed59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:10:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af0073040 con 0x7f8af00ae840 2024-03-25T18:10:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8aed59a640 10 monclient: _finish_auth 0 2024-03-25T18:10:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8aed59a640 10 monclient: _check_auth_tickets 2024-03-25T18:10:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8aed59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8af4ff6640 5 monclient: authenticate success, global_id 4355 2024-03-25T18:10:53.110 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.107+0000 7f8af4ff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.113+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8ae4001650 con 0x7f8af00ae840 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.113+0000 7f8aed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.113+0000 7f8aed59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.113+0000 7f8aed59a640 10 monclient: dump: 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:53.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.113+0000 7f8af4ff6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:10:53.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.122+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ae4001830 con 0x7f8af00ae840 2024-03-25T18:10:53.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.122+0000 7f8aed59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:53.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.122+0000 7f8aed59a640 1 -- 172.21.15.37:0/17410597 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8ae4001bc0 con 0x7f8af00ae840 2024-03-25T18:10:53.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.122+0000 7f8aed59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:53.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.122+0000 7f8aed59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:10:53.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.122+0000 7f8aed59a640 10 monclient: dump: 2024-03-25T18:10:53.124 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:53.124 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:53.124 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:53.125 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:53.125 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:53.125 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:53.125 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:53.125 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:53.125 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:53.125 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:53.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.122+0000 7f8af4ff6640 10 monclient: get_monmap_and_config success 2024-03-25T18:10:53.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.122+0000 7f8af4ff6640 4 set_mon_vals no callback set 2024-03-25T18:10:53.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.123+0000 7f8af4ff6640 10 monclient: shutdown 2024-03-25T18:10:53.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.123+0000 7f8af4ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.123+0000 7f8af4ff6640 1 -- 172.21.15.37:0/17410597 >> v1:172.21.15.37:6790/0 conn(0x7f8af00ae840 legacy=0x7f8af0145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.123+0000 7f8af4ff6640 1 -- 172.21.15.37:0/17410597 shutdown_connections 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.123+0000 7f8af4ff6640 1 -- 172.21.15.37:0/17410597 >> 172.21.15.37:0/17410597 conn(0x7f8af00b0ef0 msgr2=0x7f8af00b12d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.123+0000 7f8af4ff6640 1 -- 172.21.15.37:0/17410597 shutdown_connections 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.123+0000 7f8af4ff6640 1 -- 172.21.15.37:0/17410597 wait complete. 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.124+0000 7f8af4ff6640 10 monclient: build_initial_monmap 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.124+0000 7f8af4ff6640 10 monclient: monmap: 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:10:53.127 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:10:53.128 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:10:53.128 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:10:53.128 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:53.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.124+0000 7f8af4ff6640 1 librados: starting msgr at 2024-03-25T18:10:53.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.124+0000 7f8af4ff6640 1 librados: starting objecter 2024-03-25T18:10:53.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.124+0000 7f8af4ff6640 1 Processor -- start 2024-03-25T18:10:53.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.124+0000 7f8af4ff6640 1 -- start start 2024-03-25T18:10:53.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.124+0000 7f8af4ff6640 1 librados: setting wanted keys 2024-03-25T18:10:53.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.124+0000 7f8af4ff6640 1 librados: calling monclient init 2024-03-25T18:10:53.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.124+0000 7f8af4ff6640 10 monclient: init 2024-03-25T18:10:53.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.125+0000 7f8af4ff6640 10 monclient: _reopen_session rank -1 2024-03-25T18:10:53.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.125+0000 7f8af4ff6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:10:53.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.125+0000 7f8af4ff6640 10 monclient(hunting): picked mon.noname-b con 0x7f8af0061830 addr v1:172.21.15.117:6789/0 2024-03-25T18:10:53.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.125+0000 7f8af4ff6640 10 monclient(hunting): picked mon.noname-a con 0x7f8af00ae840 addr v1:172.21.15.37:6789/0 2024-03-25T18:10:53.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.125+0000 7f8af4ff6640 10 monclient(hunting): picked mon.noname-c con 0x7f8af013fbd0 addr v1:172.21.15.37:6790/0 2024-03-25T18:10:53.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.125+0000 7f8af4ff6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af01daa40 con 0x7f8af00ae840 2024-03-25T18:10:53.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.125+0000 7f8af4ff6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af013cc70 con 0x7f8af0061830 2024-03-25T18:10:53.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.125+0000 7f8af4ff6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8af01db670 con 0x7f8af013fbd0 2024-03-25T18:10:53.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.125+0000 7f8af4ff6640 10 monclient(hunting): _renew_subs 2024-03-25T18:10:53.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.125+0000 7f8af4ff6640 10 monclient(hunting): authenticate will time out at 7008.379057s 2024-03-25T18:10:53.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.133+0000 7f8aeed9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8af013fbd0 0x7f8af01d4540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:58904/0 (socket says 172.21.15.37:58904) 2024-03-25T18:10:53.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.134+0000 7f8aeed9d640 1 -- 172.21.15.37:0/525236355 learned_addr learned my addr 172.21.15.37:0/525236355 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:10:53.140 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.138+0000 7f8adeffd640 10 client.?.objecter ms_handle_connect 0x7f8af013fbd0 2024-03-25T18:10:53.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.138+0000 7f8adeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:53.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.139+0000 7f8adeffd640 10 client.?.objecter ms_handle_connect 0x7f8af0061830 2024-03-25T18:10:53.141 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.139+0000 7f8adeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:53.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.147+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3703252410 0 0) 0x7f8af013cc70 con 0x7f8af0061830 2024-03-25T18:10:53.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.147+0000 7f8adeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:53.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.147+0000 7f8adeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:53.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.147+0000 7f8adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:53.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.147+0000 7f8adeffd640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:10:53.149 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.147+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad0004eb0 con 0x7f8af0061830 2024-03-25T18:10:53.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.155+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612326348 0 0) 0x7f8af01db670 con 0x7f8af013fbd0 2024-03-25T18:10:53.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.155+0000 7f8adeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:53.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.155+0000 7f8adeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:53.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.155+0000 7f8adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:53.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.155+0000 7f8adeffd640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:10:53.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.155+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8af013cc70 con 0x7f8af013fbd0 2024-03-25T18:10:53.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.158+0000 7f8adeffd640 10 client.?.objecter ms_handle_connect 0x7f8af00ae840 2024-03-25T18:10:53.160 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.158+0000 7f8adeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:10:53.161 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.159+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 693585449 0 0) 0x7f8ad0004eb0 con 0x7f8af0061830 2024-03-25T18:10:53.161 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.159+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8af01db670 con 0x7f8af0061830 2024-03-25T18:10:53.161 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.159+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8ae0001810 con 0x7f8af0061830 2024-03-25T18:10:53.161 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.159+0000 7f8adeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:53.161 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.159+0000 7f8adeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.159+0000 7f8adeffd640 10 monclient(hunting): dump: 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:53.162 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.159+0000 7f8adeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:10:53.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.172+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 315466781 0 0) 0x7f8af01daa40 con 0x7f8af00ae840 2024-03-25T18:10:53.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.172+0000 7f8adeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:10:53.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.172+0000 7f8adeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:10:53.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.172+0000 7f8adeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:10:53.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.172+0000 7f8adeffd640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:10:53.174 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.172+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ad0004eb0 con 0x7f8af00ae840 2024-03-25T18:10:53.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 122501554 0 0) 0x7f8af01db670 con 0x7f8af0061830 2024-03-25T18:10:53.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 >> v1:172.21.15.37:6790/0 conn(0x7f8af013fbd0 legacy=0x7f8af01d4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:53.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 >> v1:172.21.15.37:6789/0 conn(0x7f8af00ae840 legacy=0x7f8af013d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:53.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8adeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:10:53.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8adeffd640 1 monclient: found mon.b 2024-03-25T18:10:53.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8adeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:10:53.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8adeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:53.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af01db800 con 0x7f8af0061830 2024-03-25T18:10:53.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8adeffd640 10 monclient: _finish_auth 0 2024-03-25T18:10:53.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8adeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:10:53.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8adeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:10:53.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8af4ff6640 5 monclient: authenticate success, global_id 4363 2024-03-25T18:10:53.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8af4ff6640 10 monclient: _renew_subs 2024-03-25T18:10:53.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8af4ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:53.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8af4ff6640 1 -- 172.21.15.37:0/525236355 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af01d3ec0 con 0x7f8af0061830 2024-03-25T18:10:53.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8af4ff6640 10 client.4363.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:10:53.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8af4ff6640 10 monclient: _renew_subs 2024-03-25T18:10:53.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8af4ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:53.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8af4ff6640 1 -- 172.21.15.37:0/525236355 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af013dfd0 con 0x7f8af0061830 2024-03-25T18:10:53.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.175+0000 7f8af4ff6640 1 librados: init done 2024-03-25T18:10:53.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.176+0000 7f8af4ff6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:53.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.176+0000 7f8af4ff6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:53.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.176+0000 7f8af4ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:53.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.176+0000 7f8af4ff6640 1 -- 172.21.15.37:0/525236355 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af0140a50 con 0x7f8af0061830 2024-03-25T18:10:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.457+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ae0001d20 con 0x7f8af0061830 2024-03-25T18:10:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.457+0000 7f8adeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:10:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.457+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8ae00037b0 con 0x7f8af0061830 2024-03-25T18:10:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.457+0000 7f8adeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:10:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.457+0000 7f8adeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:10:53.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.457+0000 7f8adeffd640 10 monclient: dump: 2024-03-25T18:10:53.459 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:10:53.459 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:10:53.459 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:53.459 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:10:53.459 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:10:53.460 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:10:53.460 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:10:53.460 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:10:53.460 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:10:53.460 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:10:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.457+0000 7f8aecd99640 4 set_mon_vals no callback set 2024-03-25T18:10:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.458+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f8ae0019800 con 0x7f8af0061830 2024-03-25T18:10:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.458+0000 7f8aedd9b640 10 monclient: get_auth_request con 0x7f8ad006dbb0 auth_method 0 2024-03-25T18:10:53.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.459+0000 7f8adeffd640 10 client.4363.objecter ms_handle_connect 0x7f8ad006dbb0 2024-03-25T18:10:53.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.460+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f8ae0053580 con 0x7f8af0061830 2024-03-25T18:10:53.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.460+0000 7f8adeffd640 10 client.4363.objecter ms_dispatch 0x7f8af0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:10:53.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.460+0000 7f8adeffd640 3 client.4363.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:10:53.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.460+0000 7f8adeffd640 3 client.4363.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:10:53.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.460+0000 7f8adeffd640 20 client.4363.objecter dump_active .. 0 homeless 2024-03-25T18:10:53.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.463+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ae004eec0 con 0x7f8af0061830 2024-03-25T18:10:53.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.463+0000 7f8adeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:10:53.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.463+0000 7f8adeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:10:53.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.708+0000 7f8af4ff6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:53.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.708+0000 7f8af4ff6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:53.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.708+0000 7f8af4ff6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:10:53.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.708+0000 7f8af4ff6640 1 -- 172.21.15.37:0/525236355 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8af00627b0 con 0x7f8af0061830 2024-03-25T18:10:53.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.742+0000 7f8adeffd640 1 -- 172.21.15.37:0/525236355 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f8ae0050fc0 con 0x7f8af0061830 2024-03-25T18:10:53.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.742+0000 7f8adeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:10:53.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.742+0000 7f8adeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:10:53.745 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:10:53.745 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:10:53.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 10 librados: watch_flush enter 2024-03-25T18:10:53.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 10 librados: watch_flush exit 2024-03-25T18:10:53.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 20 client.4363.objecter shutdown clearing up homeless session... 2024-03-25T18:10:53.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 10 client.4363.objecter successfully canceled tick 2024-03-25T18:10:53.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 1 -- 172.21.15.37:0/525236355 >> v1:172.21.15.117:6800/3703993921 conn(0x7f8ad006dbb0 legacy=0x7f8ad0070070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:53.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 10 monclient: shutdown 2024-03-25T18:10:53.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:10:53.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 1 -- 172.21.15.37:0/525236355 >> v1:172.21.15.117:6789/0 conn(0x7f8af0061830 legacy=0x7f8af01daf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:10:53.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 1 -- 172.21.15.37:0/525236355 shutdown_connections 2024-03-25T18:10:53.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 1 -- 172.21.15.37:0/525236355 >> 172.21.15.37:0/525236355 conn(0x7f8af00b0ef0 msgr2=0x7f8af013f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:10:53.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 1 -- 172.21.15.37:0/525236355 shutdown_connections 2024-03-25T18:10:53.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 1 -- 172.21.15.37:0/525236355 wait complete. 2024-03-25T18:10:53.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:10:53.745+0000 7f8af4ff6640 1 librados: shutdown 2024-03-25T18:10:53.760 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:00.566 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:00.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.679+0000 7f5886982640 10 monclient: get_monmap_and_config 2024-03-25T18:11:00.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.679+0000 7f5886982640 10 monclient: build_initial_monmap 2024-03-25T18:11:00.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.679+0000 7f5886982640 10 monclient: monmap: 2024-03-25T18:11:00.680 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:00.680 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:00.680 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:11:00.680554+0000 2024-03-25T18:11:00.681 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:11:00.680554+0000 2024-03-25T18:11:00.681 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:00.681 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:00.681 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:00.681 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:11:00.681 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:11:00.681 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:00.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.680+0000 7f5886982640 1 Processor -- start 2024-03-25T18:11:00.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.680+0000 7f5886982640 1 -- start start 2024-03-25T18:11:00.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.680+0000 7f5886982640 10 monclient: init 2024-03-25T18:11:00.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.681+0000 7f5886982640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:00.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.681+0000 7f5886982640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:00.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.681+0000 7f5886982640 10 monclient(hunting): picked mon.noname-a con 0x7f58800a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:00.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.681+0000 7f5886982640 10 monclient(hunting): picked mon.noname-c con 0x7f588013f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:00.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.681+0000 7f5886982640 10 monclient(hunting): picked mon.noname-b con 0x7f58801430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:00.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.681+0000 7f5886982640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5880142520 con 0x7f58800a98f0 2024-03-25T18:11:00.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.681+0000 7f5886982640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58800b28a0 con 0x7f588013f710 2024-03-25T18:11:00.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.681+0000 7f5886982640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58800b2a30 con 0x7f58801430c0 2024-03-25T18:11:00.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.681+0000 7f5886982640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:00.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.681+0000 7f5886982640 10 monclient(hunting): authenticate will time out at 7015.934251s 2024-03-25T18:11:00.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.691+0000 7f5884f1f640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f58801430c0 0x7f588014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:60414/0 (socket says 172.21.15.37:60414) 2024-03-25T18:11:00.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.691+0000 7f5884f1f640 1 -- 172.21.15.37:0/1866726058 learned_addr learned my addr 172.21.15.37:0/1866726058 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:00.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.708+0000 7f587effd640 1 -- 172.21.15.37:0/1866726058 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1769945664 0 0) 0x7f5880142520 con 0x7f58800a98f0 2024-03-25T18:11:00.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.708+0000 7f587effd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:00.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.708+0000 7f587effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:00.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.708+0000 7f587effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:00.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.708+0000 7f587effd640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:11:00.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.708+0000 7f587effd640 1 -- 172.21.15.37:0/1866726058 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5868003520 con 0x7f58800a98f0 2024-03-25T18:11:00.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.711+0000 7f587effd640 1 -- 172.21.15.37:0/1866726058 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2179038511 0 0) 0x7f58800b2a30 con 0x7f58801430c0 2024-03-25T18:11:00.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.711+0000 7f587effd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:00.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.711+0000 7f587effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:00.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.711+0000 7f587effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:00.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.711+0000 7f587effd640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:11:00.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.711+0000 7f587effd640 1 -- 172.21.15.37:0/1866726058 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5880142520 con 0x7f58801430c0 2024-03-25T18:11:00.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 1 -- 172.21.15.37:0/1866726058 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3927455991 0 0) 0x7f5868003520 con 0x7f58800a98f0 2024-03-25T18:11:00.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 1 -- 172.21.15.37:0/1866726058 >> v1:172.21.15.37:6790/0 conn(0x7f58801430c0 legacy=0x7f588014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:00.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 1 -- 172.21.15.37:0/1866726058 >> v1:172.21.15.117:6789/0 conn(0x7f588013f710 legacy=0x7f58801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:00.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:00.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 1 monclient: found mon.noname-a 2024-03-25T18:11:00.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 1 -- 172.21.15.37:0/1866726058 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58800b44c0 con 0x7f58800a98f0 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 10 monclient: _finish_auth 0 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 1 -- 172.21.15.37:0/1866726058 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5870001690 con 0x7f58800a98f0 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f587effd640 10 monclient: dump: 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:00.722 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:00.723 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:00.723 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:00.723 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:00.723 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:00.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f5886982640 5 monclient: authenticate success, global_id 4365 2024-03-25T18:11:00.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.720+0000 7f5886982640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:00.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f587effd640 1 -- 172.21.15.37:0/1866726058 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58700017f0 con 0x7f58800a98f0 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f587effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f587effd640 1 -- 172.21.15.37:0/1866726058 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5870001c00 con 0x7f58800a98f0 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f587effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f587effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f587effd640 10 monclient: dump: 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:00.737 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:00.738 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:00.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f5886982640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:00.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f5886982640 4 set_mon_vals no callback set 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f5886982640 10 monclient: shutdown 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f5886982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f5886982640 1 -- 172.21.15.37:0/1866726058 >> v1:172.21.15.37:6789/0 conn(0x7f58800a98f0 legacy=0x7f58800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f5886982640 1 -- 172.21.15.37:0/1866726058 shutdown_connections 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f5886982640 1 -- 172.21.15.37:0/1866726058 >> 172.21.15.37:0/1866726058 conn(0x7f58800b7ea0 msgr2=0x7f58800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.736+0000 7f5886982640 1 -- 172.21.15.37:0/1866726058 shutdown_connections 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.737+0000 7f5886982640 1 -- 172.21.15.37:0/1866726058 wait complete. 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.737+0000 7f5886982640 10 monclient: build_initial_monmap 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.737+0000 7f5886982640 10 monclient: monmap: 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:11:00.740 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.738+0000 7f5886982640 1 librados: starting msgr at 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.738+0000 7f5886982640 1 librados: starting objecter 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.738+0000 7f5886982640 1 Processor -- start 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.738+0000 7f5886982640 1 -- start start 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.738+0000 7f5886982640 1 librados: setting wanted keys 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.738+0000 7f5886982640 1 librados: calling monclient init 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.738+0000 7f5886982640 10 monclient: init 2024-03-25T18:11:00.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.739+0000 7f5886982640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:00.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.739+0000 7f5886982640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:00.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.739+0000 7f5886982640 10 monclient(hunting): picked mon.noname-c con 0x7f58800a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:00.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.739+0000 7f5886982640 10 monclient(hunting): picked mon.noname-a con 0x7f588013f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:00.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.739+0000 7f5886982640 10 monclient(hunting): picked mon.noname-b con 0x7f58801430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:00.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.739+0000 7f5886982640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58800b1750 con 0x7f588013f710 2024-03-25T18:11:00.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.739+0000 7f5886982640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58801399e0 con 0x7f58801430c0 2024-03-25T18:11:00.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.739+0000 7f5886982640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58800ab950 con 0x7f58800a98f0 2024-03-25T18:11:00.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.739+0000 7f5886982640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:00.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.739+0000 7f5886982640 10 monclient(hunting): authenticate will time out at 7015.991851s 2024-03-25T18:11:00.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.746+0000 7f5884f1f640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f58801430c0 0x7f58801e24a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:59670/0 (socket says 172.21.15.37:59670) 2024-03-25T18:11:00.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.746+0000 7f5884f1f640 1 -- 172.21.15.37:0/366950098 learned_addr learned my addr 172.21.15.37:0/366950098 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:00.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.747+0000 7f587cff9640 10 client.?.objecter ms_handle_connect 0x7f58800a98f0 2024-03-25T18:11:00.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.747+0000 7f587cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:00.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.749+0000 7f587cff9640 10 client.?.objecter ms_handle_connect 0x7f58801430c0 2024-03-25T18:11:00.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.749+0000 7f587cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.752+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2314845915 0 0) 0x7f58800ab950 con 0x7f58800a98f0 2024-03-25T18:11:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.752+0000 7f587cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.752+0000 7f587cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.752+0000 7f587cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.752+0000 7f587cff9640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:11:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.752+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f586c004eb0 con 0x7f58800a98f0 2024-03-25T18:11:00.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.759+0000 7f587cff9640 10 client.?.objecter ms_handle_connect 0x7f588013f710 2024-03-25T18:11:00.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.759+0000 7f587cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:00.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.761+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1111378139 0 0) 0x7f58801399e0 con 0x7f58801430c0 2024-03-25T18:11:00.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.761+0000 7f587cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:11:00.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.761+0000 7f587cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:00.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.761+0000 7f587cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:00.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.761+0000 7f587cff9640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:11:00.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.761+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58800ab950 con 0x7f58801430c0 2024-03-25T18:11:00.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.767+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2596362180 0 0) 0x7f58800ab950 con 0x7f58801430c0 2024-03-25T18:11:00.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.767+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58801399e0 con 0x7f58801430c0 2024-03-25T18:11:00.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.769+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 909461599 0 0) 0x7f586c004eb0 con 0x7f58800a98f0 2024-03-25T18:11:00.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.769+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58800ab950 con 0x7f58800a98f0 2024-03-25T18:11:00.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.769+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5870001b40 con 0x7f58800a98f0 2024-03-25T18:11:00.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.769+0000 7f587cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:00.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.769+0000 7f587cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:00.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.769+0000 7f587cff9640 10 monclient(hunting): dump: 2024-03-25T18:11:00.770 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:00.770 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.769+0000 7f587cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.769+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5878001c70 con 0x7f58801430c0 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.769+0000 7f587cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.769+0000 7f587cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.769+0000 7f587cff9640 10 monclient(hunting): dump: 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:00.771 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:00.772 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:00.772 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:00.772 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:00.772 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:00.772 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:00.772 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:00.772 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 921826173 0 0) 0x7f58801399e0 con 0x7f58801430c0 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 >> v1:172.21.15.37:6790/0 conn(0x7f58800a98f0 legacy=0x7f58800ac410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 >> v1:172.21.15.37:6789/0 conn(0x7f588013f710 legacy=0x7f5880136350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f587cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f587cff9640 1 monclient: found mon.b 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f587cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f587cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58800abc30 con 0x7f58801430c0 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f587cff9640 10 monclient: _finish_auth 0 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f587cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f587cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f5886982640 5 monclient: authenticate success, global_id 4369 2024-03-25T18:11:00.784 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f5886982640 10 monclient: _renew_subs 2024-03-25T18:11:00.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f5886982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:00.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f5886982640 1 -- 172.21.15.37:0/366950098 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5880138ea0 con 0x7f58801430c0 2024-03-25T18:11:00.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f5886982640 10 client.4369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:00.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f5886982640 10 monclient: _renew_subs 2024-03-25T18:11:00.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f5886982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:00.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f5886982640 1 -- 172.21.15.37:0/366950098 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58801387c0 con 0x7f58801430c0 2024-03-25T18:11:00.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.782+0000 7f5886982640 1 librados: init done 2024-03-25T18:11:00.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.784+0000 7f5886982640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:00.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.784+0000 7f5886982640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:00.786 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.784+0000 7f5886982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:00.786 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:00.784+0000 7f5886982640 1 -- 172.21.15.37:0/366950098 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58800627b0 con 0x7f58801430c0 2024-03-25T18:11:01.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.009+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5878004b60 con 0x7f58801430c0 2024-03-25T18:11:01.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.009+0000 7f587cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:01.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.009+0000 7f587e7fc640 4 set_mon_vals no callback set 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.050+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5878004e70 con 0x7f58801430c0 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.050+0000 7f587cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.050+0000 7f587cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.050+0000 7f587cff9640 10 monclient: dump: 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:01.051 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:01.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.055+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f587801ae40 con 0x7f58801430c0 2024-03-25T18:11:01.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.056+0000 7f587ffff640 10 monclient: get_auth_request con 0x7f586c06df30 auth_method 0 2024-03-25T18:11:01.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.056+0000 7f587cff9640 10 client.4369.objecter ms_handle_connect 0x7f586c06df30 2024-03-25T18:11:01.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.059+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f5878054bc0 con 0x7f58801430c0 2024-03-25T18:11:01.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.059+0000 7f587cff9640 10 client.4369.objecter ms_dispatch 0x7f5880000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:01.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.059+0000 7f587cff9640 3 client.4369.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:01.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.059+0000 7f587cff9640 3 client.4369.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:01.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.059+0000 7f587cff9640 20 client.4369.objecter dump_active .. 0 homeless 2024-03-25T18:11:01.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.062+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5878050500 con 0x7f58801430c0 2024-03-25T18:11:01.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.062+0000 7f587cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:01.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.062+0000 7f587cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:01.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.303+0000 7f5886982640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:01.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.303+0000 7f5886982640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:01.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.303+0000 7f5886982640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:01.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.303+0000 7f5886982640 1 -- 172.21.15.37:0/366950098 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5880139030 con 0x7f58801430c0 2024-03-25T18:11:01.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.330+0000 7f587cff9640 1 -- 172.21.15.37:0/366950098 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f5878052600 con 0x7f58801430c0 2024-03-25T18:11:01.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.330+0000 7f587cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:01.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.330+0000 7f587cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:01.331 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:11:01.331 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:01.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.332+0000 7f5886982640 10 librados: watch_flush enter 2024-03-25T18:11:01.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.332+0000 7f5886982640 10 librados: watch_flush exit 2024-03-25T18:11:01.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.332+0000 7f5886982640 20 client.4369.objecter shutdown clearing up homeless session... 2024-03-25T18:11:01.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.332+0000 7f5886982640 10 client.4369.objecter successfully canceled tick 2024-03-25T18:11:01.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.333+0000 7f5886982640 1 -- 172.21.15.37:0/366950098 >> v1:172.21.15.117:6800/3703993921 conn(0x7f586c06df30 legacy=0x7f586c0703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:01.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.333+0000 7f5886982640 10 monclient: shutdown 2024-03-25T18:11:01.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.333+0000 7f5886982640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:01.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.333+0000 7f5886982640 1 -- 172.21.15.37:0/366950098 >> v1:172.21.15.117:6789/0 conn(0x7f58801430c0 legacy=0x7f58801e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:01.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.333+0000 7f5886982640 1 -- 172.21.15.37:0/366950098 shutdown_connections 2024-03-25T18:11:01.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.333+0000 7f5886982640 1 -- 172.21.15.37:0/366950098 >> 172.21.15.37:0/366950098 conn(0x7f58800b7ea0 msgr2=0x7f5880074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:01.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.333+0000 7f5886982640 1 -- 172.21.15.37:0/366950098 shutdown_connections 2024-03-25T18:11:01.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.333+0000 7f5886982640 1 -- 172.21.15.37:0/366950098 wait complete. 2024-03-25T18:11:01.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:01.333+0000 7f5886982640 1 librados: shutdown 2024-03-25T18:11:01.346 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:04.634 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:03:39.979 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:04.641 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:03:39.987 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:04.677 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:03:40.022 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:04.707 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:03:40.020 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:04.708 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:03:40.054 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:04.717 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:03:40.030 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:04.718 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:03:40.063 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:04.749 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:03:40.094 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:04.749 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:03:40.063 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:08.151 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:08.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.268+0000 7fe8991cd640 10 monclient: get_monmap_and_config 2024-03-25T18:11:08.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.268+0000 7fe8991cd640 10 monclient: build_initial_monmap 2024-03-25T18:11:08.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.268+0000 7fe8991cd640 10 monclient: monmap: 2024-03-25T18:11:08.269 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:08.269 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:08.270 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:11:08.269609+0000 2024-03-25T18:11:08.270 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:11:08.269609+0000 2024-03-25T18:11:08.270 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:08.270 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:08.270 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:08.270 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:11:08.270 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:11:08.270 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:08.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.269+0000 7fe8991cd640 1 Processor -- start 2024-03-25T18:11:08.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.269+0000 7fe8991cd640 1 -- start start 2024-03-25T18:11:08.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.269+0000 7fe8991cd640 10 monclient: init 2024-03-25T18:11:08.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.270+0000 7fe8991cd640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:08.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.270+0000 7fe8991cd640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:08.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.270+0000 7fe8991cd640 10 monclient(hunting): picked mon.noname-a con 0x7fe8940a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:08.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.270+0000 7fe8991cd640 10 monclient(hunting): picked mon.noname-c con 0x7fe89413f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:08.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.270+0000 7fe8991cd640 10 monclient(hunting): picked mon.noname-b con 0x7fe8941430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:08.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.270+0000 7fe8991cd640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe894142520 con 0x7fe8940a98f0 2024-03-25T18:11:08.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.270+0000 7fe8991cd640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8940b28a0 con 0x7fe89413f710 2024-03-25T18:11:08.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.270+0000 7fe8991cd640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8940b2a30 con 0x7fe8941430c0 2024-03-25T18:11:08.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.270+0000 7fe8991cd640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:08.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.270+0000 7fe8991cd640 10 monclient(hunting): authenticate will time out at 7023.522619s 2024-03-25T18:11:08.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.279+0000 7fe892d9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe8940a98f0 0x7fe8940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52410/0 (socket says 172.21.15.37:52410) 2024-03-25T18:11:08.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.279+0000 7fe892d9d640 1 -- 172.21.15.37:0/1765076967 learned_addr learned my addr 172.21.15.37:0/1765076967 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:08.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.299+0000 7fe891d9b640 1 -- 172.21.15.37:0/1765076967 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 572625953 0 0) 0x7fe894142520 con 0x7fe8940a98f0 2024-03-25T18:11:08.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.299+0000 7fe891d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:11:08.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.299+0000 7fe891d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:08.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.299+0000 7fe891d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:08.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.299+0000 7fe891d9b640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:11:08.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.299+0000 7fe891d9b640 1 -- 172.21.15.37:0/1765076967 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe87c003520 con 0x7fe8940a98f0 2024-03-25T18:11:08.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.303+0000 7fe891d9b640 1 -- 172.21.15.37:0/1765076967 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3527757831 0 0) 0x7fe8940b2a30 con 0x7fe8941430c0 2024-03-25T18:11:08.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.303+0000 7fe891d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:11:08.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.303+0000 7fe891d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:08.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.303+0000 7fe891d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:08.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.303+0000 7fe891d9b640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:11:08.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.303+0000 7fe891d9b640 1 -- 172.21.15.37:0/1765076967 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe894142520 con 0x7fe8941430c0 2024-03-25T18:11:08.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 1 -- 172.21.15.37:0/1765076967 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2473318998 0 0) 0x7fe87c003520 con 0x7fe8940a98f0 2024-03-25T18:11:08.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 1 -- 172.21.15.37:0/1765076967 >> v1:172.21.15.37:6790/0 conn(0x7fe8941430c0 legacy=0x7fe89414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:08.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 1 -- 172.21.15.37:0/1765076967 >> v1:172.21.15.117:6789/0 conn(0x7fe89413f710 legacy=0x7fe8941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:08.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:11:08.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 1 monclient: found mon.noname-a 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 1 -- 172.21.15.37:0/1765076967 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8940b44c0 con 0x7fe8940a98f0 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 10 monclient: _finish_auth 0 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 1 -- 172.21.15.37:0/1765076967 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe884001690 con 0x7fe8940a98f0 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe8991cd640 5 monclient: authenticate success, global_id 4371 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe8991cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe891d9b640 10 monclient: dump: 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:08.312 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:08.313 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:08.313 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:08.313 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:08.313 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:08.313 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:08.313 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:08.313 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:08.313 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:08.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.310+0000 7fe8991cd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:08.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.324+0000 7fe891d9b640 1 -- 172.21.15.37:0/1765076967 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe8840017f0 con 0x7fe8940a98f0 2024-03-25T18:11:08.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.324+0000 7fe891d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:08.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.324+0000 7fe891d9b640 1 -- 172.21.15.37:0/1765076967 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe884001c00 con 0x7fe8940a98f0 2024-03-25T18:11:08.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.324+0000 7fe891d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:08.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.324+0000 7fe891d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:08.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.324+0000 7fe891d9b640 10 monclient: dump: 2024-03-25T18:11:08.325 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:08.325 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:08.325 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:08.325 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:08.325 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:08.326 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:08.326 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:08.326 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:08.326 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:08.326 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:08.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.324+0000 7fe8991cd640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:08.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.324+0000 7fe8991cd640 4 set_mon_vals no callback set 2024-03-25T18:11:08.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.325+0000 7fe8991cd640 10 monclient: shutdown 2024-03-25T18:11:08.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.325+0000 7fe8991cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:08.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.325+0000 7fe8991cd640 1 -- 172.21.15.37:0/1765076967 >> v1:172.21.15.37:6789/0 conn(0x7fe8940a98f0 legacy=0x7fe8940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.325+0000 7fe8991cd640 1 -- 172.21.15.37:0/1765076967 shutdown_connections 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.325+0000 7fe8991cd640 1 -- 172.21.15.37:0/1765076967 >> 172.21.15.37:0/1765076967 conn(0x7fe8940b7ea0 msgr2=0x7fe8940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.325+0000 7fe8991cd640 1 -- 172.21.15.37:0/1765076967 shutdown_connections 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.325+0000 7fe8991cd640 1 -- 172.21.15.37:0/1765076967 wait complete. 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.326+0000 7fe8991cd640 10 monclient: build_initial_monmap 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.326+0000 7fe8991cd640 10 monclient: monmap: 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:11:08.328 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:11:08.329 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:08.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.326+0000 7fe8991cd640 1 librados: starting msgr at 2024-03-25T18:11:08.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.326+0000 7fe8991cd640 1 librados: starting objecter 2024-03-25T18:11:08.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.326+0000 7fe8991cd640 1 Processor -- start 2024-03-25T18:11:08.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.326+0000 7fe8991cd640 1 -- start start 2024-03-25T18:11:08.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.326+0000 7fe8991cd640 1 librados: setting wanted keys 2024-03-25T18:11:08.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.326+0000 7fe8991cd640 1 librados: calling monclient init 2024-03-25T18:11:08.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.326+0000 7fe8991cd640 10 monclient: init 2024-03-25T18:11:08.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.327+0000 7fe8991cd640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:08.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.327+0000 7fe8991cd640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:08.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.327+0000 7fe8991cd640 10 monclient(hunting): picked mon.noname-b con 0x7fe8940a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:08.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.327+0000 7fe8991cd640 10 monclient(hunting): picked mon.noname-a con 0x7fe89413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:08.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.327+0000 7fe8991cd640 10 monclient(hunting): picked mon.noname-c con 0x7fe8941430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:08.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.327+0000 7fe8991cd640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe8941db2f0 con 0x7fe89413f710 2024-03-25T18:11:08.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.327+0000 7fe8991cd640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe894139180 con 0x7fe8940a98f0 2024-03-25T18:11:08.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.327+0000 7fe8991cd640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe894137670 con 0x7fe8941430c0 2024-03-25T18:11:08.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.327+0000 7fe8991cd640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:08.330 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.327+0000 7fe8991cd640 10 monclient(hunting): authenticate will time out at 7023.579854s 2024-03-25T18:11:08.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.335+0000 7fe89259c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fe89413f710 0x7fe894132f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52458/0 (socket says 172.21.15.37:52458) 2024-03-25T18:11:08.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.335+0000 7fe89259c640 1 -- 172.21.15.37:0/2275967122 learned_addr learned my addr 172.21.15.37:0/2275967122 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:08.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.336+0000 7fe87b7fe640 10 client.?.objecter ms_handle_connect 0x7fe8940a98f0 2024-03-25T18:11:08.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.336+0000 7fe87b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:08.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.337+0000 7fe87b7fe640 10 client.?.objecter ms_handle_connect 0x7fe89413f710 2024-03-25T18:11:08.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.337+0000 7fe87b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:08.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.339+0000 7fe87b7fe640 10 client.?.objecter ms_handle_connect 0x7fe8941430c0 2024-03-25T18:11:08.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.339+0000 7fe87b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:08.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.344+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 799278013 0 0) 0x7fe894139180 con 0x7fe8940a98f0 2024-03-25T18:11:08.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.344+0000 7fe87b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:08.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.344+0000 7fe87b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:08.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.344+0000 7fe87b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:08.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.344+0000 7fe87b7fe640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:11:08.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.344+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe880004eb0 con 0x7fe8940a98f0 2024-03-25T18:11:08.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.346+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146321384 0 0) 0x7fe8941db2f0 con 0x7fe89413f710 2024-03-25T18:11:08.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.346+0000 7fe87b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:08.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.346+0000 7fe87b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:08.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.346+0000 7fe87b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:08.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.346+0000 7fe87b7fe640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:11:08.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.346+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe894139180 con 0x7fe89413f710 2024-03-25T18:11:08.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2963728229 0 0) 0x7fe880004eb0 con 0x7fe8940a98f0 2024-03-25T18:11:08.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe8941db2f0 con 0x7fe8940a98f0 2024-03-25T18:11:08.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe884001bc0 con 0x7fe8940a98f0 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 10 monclient(hunting): dump: 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:08.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963470840 0 0) 0x7fe894137670 con 0x7fe8941430c0 2024-03-25T18:11:08.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:08.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:08.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:08.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 10 monclient(hunting): my global_id is 4370 2024-03-25T18:11:08.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.357+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe880004eb0 con 0x7fe8941430c0 2024-03-25T18:11:08.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.373+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1810825095 0 0) 0x7fe894139180 con 0x7fe89413f710 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.373+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe894137670 con 0x7fe89413f710 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.373+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe888001970 con 0x7fe89413f710 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.373+0000 7fe87b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.373+0000 7fe87b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.373+0000 7fe87b7fe640 10 monclient(hunting): dump: 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:08.374 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:08.375 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:08.375 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:08.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1326688688 0 0) 0x7fe8941db2f0 con 0x7fe8940a98f0 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 >> v1:172.21.15.37:6790/0 conn(0x7fe8941430c0 legacy=0x7fe8941e2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 >> v1:172.21.15.37:6789/0 conn(0x7fe89413f710 legacy=0x7fe894132f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe87b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe87b7fe640 1 monclient: found mon.b 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe87b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe87b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe894137200 con 0x7fe8940a98f0 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe87b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe87b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe87b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe8991cd640 5 monclient: authenticate success, global_id 4375 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe8991cd640 10 monclient: _renew_subs 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe8991cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe8991cd640 1 -- 172.21.15.37:0/2275967122 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8940b7290 con 0x7fe8940a98f0 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe8991cd640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe8991cd640 10 monclient: _renew_subs 2024-03-25T18:11:08.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe8991cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:08.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe8991cd640 1 -- 172.21.15.37:0/2275967122 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe894133680 con 0x7fe8940a98f0 2024-03-25T18:11:08.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.383+0000 7fe8991cd640 1 librados: init done 2024-03-25T18:11:08.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.384+0000 7fe8991cd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:08.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.384+0000 7fe8991cd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:08.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.384+0000 7fe8991cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:08.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.384+0000 7fe8991cd640 1 -- 172.21.15.37:0/2275967122 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe894137800 con 0x7fe8940a98f0 2024-03-25T18:11:08.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.669+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe884004a90 con 0x7fe8940a98f0 2024-03-25T18:11:08.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.669+0000 7fe87b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:08.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.669+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe884004da0 con 0x7fe8940a98f0 2024-03-25T18:11:08.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.669+0000 7fe87b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.669+0000 7fe87b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.669+0000 7fe89159a640 4 set_mon_vals no callback set 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.669+0000 7fe87b7fe640 10 monclient: dump: 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:08.671 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:08.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.670+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fe88401ad70 con 0x7fe8940a98f0 2024-03-25T18:11:08.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.671+0000 7fe89259c640 10 monclient: get_auth_request con 0x7fe88006dfb0 auth_method 0 2024-03-25T18:11:08.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.671+0000 7fe87b7fe640 10 client.4375.objecter ms_handle_connect 0x7fe88006dfb0 2024-03-25T18:11:08.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.676+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fe884054960 con 0x7fe8940a98f0 2024-03-25T18:11:08.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.676+0000 7fe87b7fe640 10 client.4375.objecter ms_dispatch 0x7fe894000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:08.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.676+0000 7fe87b7fe640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:08.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.676+0000 7fe87b7fe640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:08.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.676+0000 7fe87b7fe640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-25T18:11:08.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.677+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe8840502a0 con 0x7fe8940a98f0 2024-03-25T18:11:08.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.677+0000 7fe87b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:08.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.677+0000 7fe87b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:08.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.911+0000 7fe8991cd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:08.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.911+0000 7fe8991cd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:08.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.911+0000 7fe8991cd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:08.912 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.911+0000 7fe8991cd640 1 -- 172.21.15.37:0/2275967122 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8940627b0 con 0x7fe8940a98f0 2024-03-25T18:11:08.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.994+0000 7fe87b7fe640 1 -- 172.21.15.37:0/2275967122 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fe8840523a0 con 0x7fe8940a98f0 2024-03-25T18:11:08.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.994+0000 7fe87b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:08.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.994+0000 7fe87b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:08.996 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:11:08.996 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:08.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.996+0000 7fe8991cd640 10 librados: watch_flush enter 2024-03-25T18:11:08.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 10 librados: watch_flush exit 2024-03-25T18:11:08.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-25T18:11:08.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 10 client.4375.objecter successfully canceled tick 2024-03-25T18:11:08.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 1 -- 172.21.15.37:0/2275967122 >> v1:172.21.15.117:6800/3703993921 conn(0x7fe88006dfb0 legacy=0x7fe880070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:08.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 10 monclient: shutdown 2024-03-25T18:11:08.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:08.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 1 -- 172.21.15.37:0/2275967122 >> v1:172.21.15.117:6789/0 conn(0x7fe8940a98f0 legacy=0x7fe8941dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:08.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 1 -- 172.21.15.37:0/2275967122 shutdown_connections 2024-03-25T18:11:08.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 1 -- 172.21.15.37:0/2275967122 >> 172.21.15.37:0/2275967122 conn(0x7fe8940b7ea0 msgr2=0x7fe894074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:08.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 1 -- 172.21.15.37:0/2275967122 shutdown_connections 2024-03-25T18:11:08.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 1 -- 172.21.15.37:0/2275967122 wait complete. 2024-03-25T18:11:08.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:08.997+0000 7fe8991cd640 1 librados: shutdown 2024-03-25T18:11:09.010 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:15.817 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.929+0000 7fa2215d6640 10 monclient: get_monmap_and_config 2024-03-25T18:11:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.929+0000 7fa2215d6640 10 monclient: build_initial_monmap 2024-03-25T18:11:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.929+0000 7fa2215d6640 10 monclient: monmap: 2024-03-25T18:11:15.930 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:15.930 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:15.930 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:11:15.930172+0000 2024-03-25T18:11:15.930 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:11:15.930172+0000 2024-03-25T18:11:15.930 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:15.930 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:15.930 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:15.931 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:11:15.931 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:11:15.931 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.929+0000 7fa2215d6640 1 Processor -- start 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.929+0000 7fa2215d6640 1 -- start start 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.929+0000 7fa2215d6640 10 monclient: init 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.930+0000 7fa2215d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.930+0000 7fa2215d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.930+0000 7fa2215d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa21c0b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.930+0000 7fa2215d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa21c0b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.930+0000 7fa2215d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa21c14be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.930+0000 7fa2215d6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c0af940 con 0x7fa21c0b4d60 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.930+0000 7fa2215d6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c0b0a70 con 0x7fa21c0b1730 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.930+0000 7fa2215d6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c0b0d50 con 0x7fa21c14be90 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.930+0000 7fa2215d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:15.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.930+0000 7fa2215d6640 10 monclient(hunting): authenticate will time out at 7031.182848s 2024-03-25T18:11:15.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.938+0000 7fa21b7fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fa21c14be90 0x7fa21c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:58592/0 (socket says 172.21.15.37:58592) 2024-03-25T18:11:15.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.938+0000 7fa21b7fe640 1 -- 172.21.15.37:0/2864650403 learned_addr learned my addr 172.21.15.37:0/2864650403 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:15.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.948+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244931589 0 0) 0x7fa21c0b0d50 con 0x7fa21c14be90 2024-03-25T18:11:15.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.948+0000 7fa21a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:15.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.948+0000 7fa21a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:15.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.948+0000 7fa21a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:15.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.948+0000 7fa21a7fc640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:11:15.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.948+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa208003520 con 0x7fa21c14be90 2024-03-25T18:11:15.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.949+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2347340222 0 0) 0x7fa21c0b0a70 con 0x7fa21c0b1730 2024-03-25T18:11:15.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.949+0000 7fa21a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:15.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.949+0000 7fa21a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:15.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.949+0000 7fa21a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:15.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.949+0000 7fa21a7fc640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:11:15.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.949+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa21c0b0d50 con 0x7fa21c0b1730 2024-03-25T18:11:15.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.978+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 455925305 0 0) 0x7fa21c0af940 con 0x7fa21c0b4d60 2024-03-25T18:11:15.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.978+0000 7fa21a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:15.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.978+0000 7fa21a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:15.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.978+0000 7fa21a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:15.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.978+0000 7fa21a7fc640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:11:15.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.978+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa21c0b0a70 con 0x7fa21c0b4d60 2024-03-25T18:11:15.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2103124723 0 0) 0x7fa21c0b0d50 con 0x7fa21c0b1730 2024-03-25T18:11:15.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 >> v1:172.21.15.37:6790/0 conn(0x7fa21c14be90 legacy=0x7fa21c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:15.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 >> v1:172.21.15.37:6789/0 conn(0x7fa21c0b4d60 legacy=0x7fa21c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:15.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:15.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c0b1030 con 0x7fa21c0b1730 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa204001670 con 0x7fa21c0b1730 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa21a7fc640 10 monclient: dump: 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:15.982 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:15.983 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:15.983 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:15.983 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:15.983 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:15.983 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:15.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa2215d6640 5 monclient: authenticate success, global_id 4378 2024-03-25T18:11:15.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.980+0000 7fa2215d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:15.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.992+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2040017d0 con 0x7fa21c0b1730 2024-03-25T18:11:15.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.992+0000 7fa21a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:15.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.992+0000 7fa21a7fc640 1 -- 172.21.15.37:0/2864650403 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa204001ba0 con 0x7fa21c0b1730 2024-03-25T18:11:15.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.992+0000 7fa2215d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:15.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.992+0000 7fa2215d6640 4 set_mon_vals no callback set 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.992+0000 7fa21a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.992+0000 7fa21a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.992+0000 7fa21a7fc640 10 monclient: dump: 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:15.994 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:15.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.993+0000 7fa2215d6640 10 monclient: shutdown 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.993+0000 7fa2215d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.993+0000 7fa2215d6640 1 -- 172.21.15.37:0/2864650403 >> v1:172.21.15.117:6789/0 conn(0x7fa21c0b1730 legacy=0x7fa21c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.993+0000 7fa2215d6640 1 -- 172.21.15.37:0/2864650403 shutdown_connections 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.993+0000 7fa2215d6640 1 -- 172.21.15.37:0/2864650403 >> 172.21.15.37:0/2864650403 conn(0x7fa21c0617a0 msgr2=0x7fa21c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.993+0000 7fa2215d6640 1 -- 172.21.15.37:0/2864650403 shutdown_connections 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.993+0000 7fa2215d6640 1 -- 172.21.15.37:0/2864650403 wait complete. 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.994+0000 7fa2215d6640 10 monclient: build_initial_monmap 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.994+0000 7fa2215d6640 10 monclient: monmap: 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:15.996 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:11:15.997 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:11:15.997 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:15.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.994+0000 7fa2215d6640 1 librados: starting msgr at 2024-03-25T18:11:15.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.994+0000 7fa2215d6640 1 librados: starting objecter 2024-03-25T18:11:15.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.994+0000 7fa2215d6640 1 Processor -- start 2024-03-25T18:11:15.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 1 -- start start 2024-03-25T18:11:15.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 1 librados: setting wanted keys 2024-03-25T18:11:15.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 1 librados: calling monclient init 2024-03-25T18:11:15.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 10 monclient: init 2024-03-25T18:11:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:11:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa21c0b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa21c0b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa21c14be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c13a4c0 con 0x7fa21c14be90 2024-03-25T18:11:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c13a190 con 0x7fa21c0b1730 2024-03-25T18:11:15.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa21c13b6e0 con 0x7fa21c0b4d60 2024-03-25T18:11:15.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:15.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:15.995+0000 7fa2215d6640 10 monclient(hunting): authenticate will time out at 7031.248129s 2024-03-25T18:11:16.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.006+0000 7fa21affd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fa21c0b1730 0x7fa21c139a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:49212/0 (socket says 172.21.15.37:49212) 2024-03-25T18:11:16.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.006+0000 7fa21affd640 1 -- 172.21.15.37:0/480469692 learned_addr learned my addr 172.21.15.37:0/480469692 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:16.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.006+0000 7fa213fff640 10 client.?.objecter ms_handle_connect 0x7fa21c0b1730 2024-03-25T18:11:16.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.006+0000 7fa213fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:16.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.011+0000 7fa213fff640 10 client.?.objecter ms_handle_connect 0x7fa21c0b4d60 2024-03-25T18:11:16.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.011+0000 7fa213fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:16.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.019+0000 7fa213fff640 10 client.?.objecter ms_handle_connect 0x7fa21c14be90 2024-03-25T18:11:16.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.019+0000 7fa213fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:16.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.020+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2135046185 0 0) 0x7fa21c13b6e0 con 0x7fa21c0b4d60 2024-03-25T18:11:16.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.020+0000 7fa213fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:16.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.020+0000 7fa213fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:16.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.020+0000 7fa213fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:16.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.020+0000 7fa213fff640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:11:16.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.020+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa1fc004eb0 con 0x7fa21c0b4d60 2024-03-25T18:11:16.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.028+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157486990 0 0) 0x7fa21c13a4c0 con 0x7fa21c14be90 2024-03-25T18:11:16.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.028+0000 7fa213fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:16.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.028+0000 7fa213fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:16.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.028+0000 7fa213fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:16.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.028+0000 7fa213fff640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:11:16.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.028+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa21c13b6e0 con 0x7fa21c14be90 2024-03-25T18:11:16.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.029+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455793314 0 0) 0x7fa21c13a190 con 0x7fa21c0b1730 2024-03-25T18:11:16.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.029+0000 7fa213fff640 10 monclient(hunting): _negotiate 2024-03-25T18:11:16.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.029+0000 7fa213fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:16.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.029+0000 7fa213fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:16.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.029+0000 7fa213fff640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:11:16.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.029+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa21c13a4c0 con 0x7fa21c0b1730 2024-03-25T18:11:16.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.035+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2783453366 0 0) 0x7fa1fc004eb0 con 0x7fa21c0b4d60 2024-03-25T18:11:16.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.035+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa21c13a190 con 0x7fa21c0b4d60 2024-03-25T18:11:16.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.035+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa204001ae0 con 0x7fa21c0b4d60 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.035+0000 7fa213fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.035+0000 7fa213fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.035+0000 7fa213fff640 10 monclient(hunting): dump: 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:16.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.036+0000 7fa213fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:16.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.047+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2297958027 0 0) 0x7fa21c13a4c0 con 0x7fa21c0b1730 2024-03-25T18:11:16.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.047+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa1fc004eb0 con 0x7fa21c0b1730 2024-03-25T18:11:16.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.047+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa20c001a40 con 0x7fa21c0b1730 2024-03-25T18:11:16.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.047+0000 7fa213fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:16.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.047+0000 7fa213fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:16.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.047+0000 7fa213fff640 10 monclient(hunting): dump: 2024-03-25T18:11:16.049 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:16.049 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:16.049 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:16.049 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:16.049 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:16.049 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:16.049 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:16.049 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:16.049 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:16.049 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3816819038 0 0) 0x7fa21c13a190 con 0x7fa21c0b4d60 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 >> v1:172.21.15.117:6789/0 conn(0x7fa21c0b1730 legacy=0x7fa21c139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 >> v1:172.21.15.37:6789/0 conn(0x7fa21c14be90 legacy=0x7fa21c1e1fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa213fff640 10 monclient: _finish_hunting 0 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa213fff640 1 monclient: found mon.c 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa213fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa213fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21c139010 con 0x7fa21c0b4d60 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa213fff640 10 monclient: _finish_auth 0 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa213fff640 10 monclient: _check_auth_tickets 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa213fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa2215d6640 5 monclient: authenticate success, global_id 4376 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa2215d6640 10 monclient: _renew_subs 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa2215d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.048+0000 7fa2215d6640 1 -- 172.21.15.37:0/480469692 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa21c13bb00 con 0x7fa21c0b4d60 2024-03-25T18:11:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.049+0000 7fa2215d6640 10 client.4376.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:16.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.049+0000 7fa2215d6640 10 monclient: _renew_subs 2024-03-25T18:11:16.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.049+0000 7fa2215d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:16.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.049+0000 7fa2215d6640 1 -- 172.21.15.37:0/480469692 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa21c1d80f0 con 0x7fa21c0b4d60 2024-03-25T18:11:16.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.049+0000 7fa2215d6640 1 librados: init done 2024-03-25T18:11:16.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.050+0000 7fa2215d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:16.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.050+0000 7fa2215d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:16.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.050+0000 7fa2215d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:16.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.050+0000 7fa2215d6640 1 -- 172.21.15.37:0/480469692 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa21c0afbd0 con 0x7fa21c0b4d60 2024-03-25T18:11:16.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.379+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2040038a0 con 0x7fa21c0b4d60 2024-03-25T18:11:16.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.379+0000 7fa213fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:16.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.379+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa204003bb0 con 0x7fa21c0b4d60 2024-03-25T18:11:16.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.379+0000 7fa213fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:16.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.379+0000 7fa213fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:16.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.379+0000 7fa213fff640 10 monclient: dump: 2024-03-25T18:11:16.381 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:16.381 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:16.381 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:16.382 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:16.382 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:16.382 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:16.382 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:16.382 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:16.382 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:16.382 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:16.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.379+0000 7fa219ffb640 4 set_mon_vals no callback set 2024-03-25T18:11:16.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.380+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fa204019b80 con 0x7fa21c0b4d60 2024-03-25T18:11:16.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.380+0000 7fa21affd640 10 monclient: get_auth_request con 0x7fa1fc06df60 auth_method 0 2024-03-25T18:11:16.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.381+0000 7fa213fff640 10 client.4376.objecter ms_handle_connect 0x7fa1fc06df60 2024-03-25T18:11:16.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.386+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fa204053680 con 0x7fa21c0b4d60 2024-03-25T18:11:16.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.386+0000 7fa213fff640 10 client.4376.objecter ms_dispatch 0x7fa21c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:16.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.386+0000 7fa213fff640 3 client.4376.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:16.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.386+0000 7fa213fff640 3 client.4376.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:16.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.386+0000 7fa213fff640 20 client.4376.objecter dump_active .. 0 homeless 2024-03-25T18:11:16.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.386+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa20404f190 con 0x7fa21c0b4d60 2024-03-25T18:11:16.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.386+0000 7fa213fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:16.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.386+0000 7fa213fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:16.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.611+0000 7fa2215d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:16.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.611+0000 7fa2215d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:16.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.611+0000 7fa2215d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:16.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.611+0000 7fa2215d6640 1 -- 172.21.15.37:0/480469692 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa21c1d7520 con 0x7fa21c0b4d60 2024-03-25T18:11:16.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.642+0000 7fa213fff640 1 -- 172.21.15.37:0/480469692 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fa21c1d7520 con 0x7fa21c0b4d60 2024-03-25T18:11:16.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.642+0000 7fa213fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:16.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.642+0000 7fa213fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:16.644 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:11:16.644 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:16.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 10 librados: watch_flush enter 2024-03-25T18:11:16.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 10 librados: watch_flush exit 2024-03-25T18:11:16.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 20 client.4376.objecter shutdown clearing up homeless session... 2024-03-25T18:11:16.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 10 client.4376.objecter successfully canceled tick 2024-03-25T18:11:16.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 1 -- 172.21.15.37:0/480469692 >> v1:172.21.15.117:6800/3703993921 conn(0x7fa1fc06df60 legacy=0x7fa1fc070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:16.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 10 monclient: shutdown 2024-03-25T18:11:16.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:16.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 1 -- 172.21.15.37:0/480469692 >> v1:172.21.15.37:6790/0 conn(0x7fa21c0b4d60 legacy=0x7fa21c1d7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:16.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 1 -- 172.21.15.37:0/480469692 shutdown_connections 2024-03-25T18:11:16.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 1 -- 172.21.15.37:0/480469692 >> 172.21.15.37:0/480469692 conn(0x7fa21c0617a0 msgr2=0x7fa21c0b2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:16.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 1 -- 172.21.15.37:0/480469692 shutdown_connections 2024-03-25T18:11:16.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 1 -- 172.21.15.37:0/480469692 wait complete. 2024-03-25T18:11:16.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:16.645+0000 7fa2215d6640 1 librados: shutdown 2024-03-25T18:11:16.658 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:23.465 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:23.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.584+0000 7f9dd3d93640 10 monclient: get_monmap_and_config 2024-03-25T18:11:23.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.584+0000 7f9dd3d93640 10 monclient: build_initial_monmap 2024-03-25T18:11:23.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.584+0000 7f9dd3d93640 10 monclient: monmap: 2024-03-25T18:11:23.585 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:23.585 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:23.585 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:11:23.585290+0000 2024-03-25T18:11:23.585 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:11:23.585290+0000 2024-03-25T18:11:23.586 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:23.586 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:23.586 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:23.586 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:11:23.586 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:11:23.586 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:23.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.585+0000 7f9dd3d93640 1 Processor -- start 2024-03-25T18:11:23.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.585+0000 7f9dd3d93640 1 -- start start 2024-03-25T18:11:23.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.585+0000 7f9dd3d93640 10 monclient: init 2024-03-25T18:11:23.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.586+0000 7f9dd3d93640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:23.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.586+0000 7f9dd3d93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:23.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.586+0000 7f9dd3d93640 10 monclient(hunting): picked mon.noname-b con 0x7f9dcc0a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:23.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.586+0000 7f9dd3d93640 10 monclient(hunting): picked mon.noname-a con 0x7f9dcc13f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:23.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.586+0000 7f9dd3d93640 10 monclient(hunting): picked mon.noname-c con 0x7f9dcc1430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:23.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.586+0000 7f9dd3d93640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc142520 con 0x7f9dcc13f710 2024-03-25T18:11:23.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.586+0000 7f9dd3d93640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc0b28a0 con 0x7f9dcc1430c0 2024-03-25T18:11:23.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.586+0000 7f9dd3d93640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc0b2a30 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.586+0000 7f9dd3d93640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:23.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.586+0000 7f9dd3d93640 10 monclient(hunting): authenticate will time out at 7038.838746s 2024-03-25T18:11:23.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.595+0000 7f9dd132e640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f9dcc13f710 0x7f9dcc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:47758/0 (socket says 172.21.15.37:47758) 2024-03-25T18:11:23.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.595+0000 7f9dd132e640 1 -- 172.21.15.37:0/1709729794 learned_addr learned my addr 172.21.15.37:0/1709729794 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:23.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.611+0000 7f9dd0b2d640 1 -- 172.21.15.37:0/1709729794 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4073669022 0 0) 0x7f9dcc0b28a0 con 0x7f9dcc1430c0 2024-03-25T18:11:23.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.611+0000 7f9dd0b2d640 10 monclient(hunting): _negotiate 2024-03-25T18:11:23.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.611+0000 7f9dd0b2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:23.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.611+0000 7f9dd0b2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:23.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.611+0000 7f9dd0b2d640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:11:23.612 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.611+0000 7f9dd0b2d640 1 -- 172.21.15.37:0/1709729794 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9db8003520 con 0x7f9dcc1430c0 2024-03-25T18:11:23.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.614+0000 7f9dd0b2d640 1 -- 172.21.15.37:0/1709729794 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 819693758 0 0) 0x7f9dcc0b2a30 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.614+0000 7f9dd0b2d640 10 monclient(hunting): _negotiate 2024-03-25T18:11:23.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.614+0000 7f9dd0b2d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:23.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.614+0000 7f9dd0b2d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:23.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.614+0000 7f9dd0b2d640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:11:23.615 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.614+0000 7f9dd0b2d640 1 -- 172.21.15.37:0/1709729794 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc0b28a0 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 1 -- 172.21.15.37:0/1709729794 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4044673725 0 0) 0x7f9db8003520 con 0x7f9dcc1430c0 2024-03-25T18:11:23.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 1 -- 172.21.15.37:0/1709729794 >> v1:172.21.15.37:6790/0 conn(0x7f9dcc0a98f0 legacy=0x7f9dcc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:23.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 1 -- 172.21.15.37:0/1709729794 >> v1:172.21.15.37:6789/0 conn(0x7f9dcc13f710 legacy=0x7f9dcc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:23.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 10 monclient: _finish_hunting 0 2024-03-25T18:11:23.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 1 monclient: found mon.noname-c 2024-03-25T18:11:23.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:23.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:11:23.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 1 -- 172.21.15.37:0/1709729794 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc0b44c0 con 0x7f9dcc1430c0 2024-03-25T18:11:23.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 10 monclient: _finish_auth 0 2024-03-25T18:11:23.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 10 monclient: _check_auth_tickets 2024-03-25T18:11:23.619 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 1 -- 172.21.15.37:0/1709729794 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9dc4001a10 con 0x7f9dcc1430c0 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd0b2d640 10 monclient: dump: 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd3d93640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:11:23.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.617+0000 7f9dd3d93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:23.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.645+0000 7f9dd0b2d640 1 -- 172.21.15.37:0/1709729794 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dc4001b70 con 0x7f9dcc1430c0 2024-03-25T18:11:23.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.645+0000 7f9dd0b2d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:23.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.645+0000 7f9dd0b2d640 1 -- 172.21.15.37:0/1709729794 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9dc4001f80 con 0x7f9dcc1430c0 2024-03-25T18:11:23.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.645+0000 7f9dd0b2d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:23.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.645+0000 7f9dd0b2d640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:23.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.645+0000 7f9dd0b2d640 10 monclient: dump: 2024-03-25T18:11:23.646 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:23.646 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:23.647 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:23.647 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:23.647 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:23.647 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:23.647 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:23.647 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:23.647 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:23.647 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:23.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.645+0000 7f9dd3d93640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:23.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.645+0000 7f9dd3d93640 4 set_mon_vals no callback set 2024-03-25T18:11:23.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.646+0000 7f9dd3d93640 10 monclient: shutdown 2024-03-25T18:11:23.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.646+0000 7f9dd3d93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:23.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.646+0000 7f9dd3d93640 1 -- 172.21.15.37:0/1709729794 >> v1:172.21.15.117:6789/0 conn(0x7f9dcc1430c0 legacy=0x7f9dcc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.646+0000 7f9dd3d93640 1 -- 172.21.15.37:0/1709729794 shutdown_connections 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.646+0000 7f9dd3d93640 1 -- 172.21.15.37:0/1709729794 >> 172.21.15.37:0/1709729794 conn(0x7f9dcc0b7ea0 msgr2=0x7f9dcc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.646+0000 7f9dd3d93640 1 -- 172.21.15.37:0/1709729794 shutdown_connections 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.646+0000 7f9dd3d93640 1 -- 172.21.15.37:0/1709729794 wait complete. 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.647+0000 7f9dd3d93640 10 monclient: build_initial_monmap 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.647+0000 7f9dd3d93640 10 monclient: monmap: 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:11:23.650 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.647+0000 7f9dd3d93640 1 librados: starting msgr at 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.647+0000 7f9dd3d93640 1 librados: starting objecter 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.647+0000 7f9dd3d93640 1 Processor -- start 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.647+0000 7f9dd3d93640 1 -- start start 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.647+0000 7f9dd3d93640 1 librados: setting wanted keys 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.647+0000 7f9dd3d93640 1 librados: calling monclient init 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.647+0000 7f9dd3d93640 10 monclient: init 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.648+0000 7f9dd3d93640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.648+0000 7f9dd3d93640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.648+0000 7f9dd3d93640 10 monclient(hunting): picked mon.noname-a con 0x7f9dcc0a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.648+0000 7f9dd3d93640 10 monclient(hunting): picked mon.noname-c con 0x7f9dcc13f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.648+0000 7f9dd3d93640 10 monclient(hunting): picked mon.noname-b con 0x7f9dcc1430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.648+0000 7f9dd3d93640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc136a10 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.648+0000 7f9dd3d93640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc133b60 con 0x7f9dcc1430c0 2024-03-25T18:11:23.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.648+0000 7f9dd3d93640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9dcc0ab950 con 0x7f9dcc13f710 2024-03-25T18:11:23.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.648+0000 7f9dd3d93640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:23.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.648+0000 7f9dd3d93640 10 monclient(hunting): authenticate will time out at 7038.900903s 2024-03-25T18:11:23.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.651+0000 7f9dd2330640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f9dcc1430c0 0x7f9dcc133450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:36666/0 (socket says 172.21.15.37:36666) 2024-03-25T18:11:23.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.651+0000 7f9dd2330640 1 -- 172.21.15.37:0/2739656642 learned_addr learned my addr 172.21.15.37:0/2739656642 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:23.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.653+0000 7f9dc27fc640 10 client.?.objecter ms_handle_connect 0x7f9dcc1430c0 2024-03-25T18:11:23.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.654+0000 7f9dc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:23.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.661+0000 7f9dc27fc640 10 client.?.objecter ms_handle_connect 0x7f9dcc13f710 2024-03-25T18:11:23.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.661+0000 7f9dc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:23.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.666+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803328724 0 0) 0x7f9dcc133b60 con 0x7f9dcc1430c0 2024-03-25T18:11:23.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.666+0000 7f9dc27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:23.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.666+0000 7f9dc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:23.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.666+0000 7f9dc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:23.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.666+0000 7f9dc27fc640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:11:23.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.666+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9db0004eb0 con 0x7f9dcc1430c0 2024-03-25T18:11:23.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.667+0000 7f9dc27fc640 10 client.?.objecter ms_handle_connect 0x7f9dcc0a98f0 2024-03-25T18:11:23.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.667+0000 7f9dc27fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:23.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.677+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705082738 0 0) 0x7f9dcc136a10 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.677+0000 7f9dc27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:23.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.677+0000 7f9dc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:23.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.677+0000 7f9dc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:23.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.677+0000 7f9dc27fc640 10 monclient(hunting): my global_id is 4386 2024-03-25T18:11:23.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.677+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc133b60 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.684+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2906870232 0 0) 0x7f9dcc0ab950 con 0x7f9dcc13f710 2024-03-25T18:11:23.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.684+0000 7f9dc27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:23.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.684+0000 7f9dc27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:23.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.684+0000 7f9dc27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:23.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.684+0000 7f9dc27fc640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:11:23.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.684+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9dcc136a10 con 0x7f9dcc13f710 2024-03-25T18:11:23.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.691+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 268258363 0 0) 0x7f9dcc133b60 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.691+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dcc0ab950 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.691+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9dc8001930 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.691+0000 7f9dc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.691+0000 7f9dc27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.691+0000 7f9dc27fc640 10 monclient(hunting): dump: 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:23.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.691+0000 7f9dc27fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:23.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.692+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1540328333 0 0) 0x7f9db0004eb0 con 0x7f9dcc1430c0 2024-03-25T18:11:23.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.692+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9dcc133b60 con 0x7f9dcc1430c0 2024-03-25T18:11:23.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.692+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9dc4001f80 con 0x7f9dcc1430c0 2024-03-25T18:11:23.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.692+0000 7f9dc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:23.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.692+0000 7f9dc27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:23.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.692+0000 7f9dc27fc640 10 monclient(hunting): dump: 2024-03-25T18:11:23.694 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:23.694 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:23.694 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:23.695 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:23.695 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:23.695 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:23.695 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:23.695 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:23.695 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:23.695 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:23.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.704+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1048705045 0 0) 0x7f9dcc136a10 con 0x7f9dcc13f710 2024-03-25T18:11:23.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.704+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9db0004eb0 con 0x7f9dcc13f710 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.704+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9dbc001860 con 0x7f9dcc13f710 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.704+0000 7f9dc27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.704+0000 7f9dc27fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.704+0000 7f9dc27fc640 10 monclient(hunting): dump: 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:23.706 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:23.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1176399984 0 0) 0x7f9dcc0ab950 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 >> v1:172.21.15.37:6790/0 conn(0x7f9dcc13f710 legacy=0x7f9dcc136300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:23.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 >> v1:172.21.15.117:6789/0 conn(0x7f9dcc1430c0 legacy=0x7f9dcc133450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:23.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dc27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:23.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dc27fc640 1 monclient: found mon.a 2024-03-25T18:11:23.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dc27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:23.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dc27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:11:23.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dcc0abc30 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dc27fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:23.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dc27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:23.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dc27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:23.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dd3d93640 5 monclient: authenticate success, global_id 4386 2024-03-25T18:11:23.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dd3d93640 10 monclient: _renew_subs 2024-03-25T18:11:23.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dd3d93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:11:23.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.714+0000 7f9dd3d93640 1 -- 172.21.15.37:0/2739656642 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dcc0abf10 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.715+0000 7f9dd3d93640 10 client.4386.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:23.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.715+0000 7f9dd3d93640 10 monclient: _renew_subs 2024-03-25T18:11:23.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.715+0000 7f9dd3d93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:11:23.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.715+0000 7f9dd3d93640 1 -- 172.21.15.37:0/2739656642 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dcc137e00 con 0x7f9dcc0a98f0 2024-03-25T18:11:23.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.715+0000 7f9dd3d93640 1 librados: init done 2024-03-25T18:11:23.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.716+0000 7f9dd3d93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:23.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.716+0000 7f9dd3d93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:23.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.716+0000 7f9dd3d93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:11:23.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:23.716+0000 7f9dd3d93640 1 -- 172.21.15.37:0/2739656642 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dcc0627b0 con 0x7f9dcc0a98f0 2024-03-25T18:11:24.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.045+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9dc8003610 con 0x7f9dcc0a98f0 2024-03-25T18:11:24.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.045+0000 7f9dc27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:24.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.045+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9dc8003920 con 0x7f9dcc0a98f0 2024-03-25T18:11:24.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.045+0000 7f9dc27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:24.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.045+0000 7f9dc27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:24.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.045+0000 7f9dc27fc640 10 monclient: dump: 2024-03-25T18:11:24.047 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:24.047 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:24.047 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:24.047 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:24.047 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:24.048 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:24.048 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:24.048 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:24.048 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:24.048 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:24.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.045+0000 7f9dc3fff640 4 set_mon_vals no callback set 2024-03-25T18:11:24.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.046+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f9dc80198f0 con 0x7f9dcc0a98f0 2024-03-25T18:11:24.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.047+0000 7f9dd132e640 10 monclient: get_auth_request con 0x7f9db006e0b0 auth_method 0 2024-03-25T18:11:24.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.047+0000 7f9dc27fc640 10 client.4386.objecter ms_handle_connect 0x7f9db006e0b0 2024-03-25T18:11:24.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.053+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f9dc8053560 con 0x7f9dcc0a98f0 2024-03-25T18:11:24.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.053+0000 7f9dc27fc640 10 client.4386.objecter ms_dispatch 0x7f9dcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:24.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.053+0000 7f9dc27fc640 3 client.4386.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:24.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.053+0000 7f9dc27fc640 3 client.4386.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:24.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.053+0000 7f9dc27fc640 20 client.4386.objecter dump_active .. 0 homeless 2024-03-25T18:11:24.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.053+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9dc804eea0 con 0x7f9dcc0a98f0 2024-03-25T18:11:24.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.053+0000 7f9dc27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:24.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.053+0000 7f9dc27fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:24.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.282+0000 7f9dd3d93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:24.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.282+0000 7f9dd3d93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:24.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.282+0000 7f9dd3d93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:11:24.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.282+0000 7f9dd3d93640 1 -- 172.21.15.37:0/2739656642 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9dcc13c060 con 0x7f9dcc0a98f0 2024-03-25T18:11:24.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.312+0000 7f9dc27fc640 1 -- 172.21.15.37:0/2739656642 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f9dc8050fa0 con 0x7f9dcc0a98f0 2024-03-25T18:11:24.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.312+0000 7f9dc27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:24.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.312+0000 7f9dc27fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:24.314 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:11:24.314 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:24.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.314+0000 7f9dd3d93640 10 librados: watch_flush enter 2024-03-25T18:11:24.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.314+0000 7f9dd3d93640 10 librados: watch_flush exit 2024-03-25T18:11:24.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.314+0000 7f9dd3d93640 20 client.4386.objecter shutdown clearing up homeless session... 2024-03-25T18:11:24.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.314+0000 7f9dd3d93640 10 client.4386.objecter successfully canceled tick 2024-03-25T18:11:24.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.314+0000 7f9dd3d93640 1 -- 172.21.15.37:0/2739656642 >> v1:172.21.15.117:6800/3703993921 conn(0x7f9db006e0b0 legacy=0x7f9db0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:24.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.314+0000 7f9dd3d93640 10 monclient: shutdown 2024-03-25T18:11:24.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.314+0000 7f9dd3d93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:24.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.314+0000 7f9dd3d93640 1 -- 172.21.15.37:0/2739656642 >> v1:172.21.15.37:6789/0 conn(0x7f9dcc0a98f0 legacy=0x7f9dcc132d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:24.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.315+0000 7f9dd3d93640 1 -- 172.21.15.37:0/2739656642 shutdown_connections 2024-03-25T18:11:24.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.315+0000 7f9dd3d93640 1 -- 172.21.15.37:0/2739656642 >> 172.21.15.37:0/2739656642 conn(0x7f9dcc0b7ea0 msgr2=0x7f9dcc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:24.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.315+0000 7f9dd3d93640 1 -- 172.21.15.37:0/2739656642 shutdown_connections 2024-03-25T18:11:24.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.315+0000 7f9dd3d93640 1 -- 172.21.15.37:0/2739656642 wait complete. 2024-03-25T18:11:24.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:24.315+0000 7f9dd3d93640 1 librados: shutdown 2024-03-25T18:11:24.328 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:29.697 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:04:05.042 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:29.700 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:04:05.046 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:29.733 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:04:05.078 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:29.752 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:04:05.065 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:29.759 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:04:05.105 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:29.766 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:04:05.079 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:29.768 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:04:05.114 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:29.792 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:04:05.106 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:29.804 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:04:05.149 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:11:31.134 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:31.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.245+0000 7f47e8389640 10 monclient: get_monmap_and_config 2024-03-25T18:11:31.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.245+0000 7f47e8389640 10 monclient: build_initial_monmap 2024-03-25T18:11:31.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.245+0000 7f47e8389640 10 monclient: monmap: 2024-03-25T18:11:31.247 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:31.247 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:31.247 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:11:31.246687+0000 2024-03-25T18:11:31.247 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:11:31.246687+0000 2024-03-25T18:11:31.247 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:31.247 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:31.247 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:31.247 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:11:31.247 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:11:31.247 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:31.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.246+0000 7f47e8389640 1 Processor -- start 2024-03-25T18:11:31.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.246+0000 7f47e8389640 1 -- start start 2024-03-25T18:11:31.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.246+0000 7f47e8389640 10 monclient: init 2024-03-25T18:11:31.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.247+0000 7f47e8389640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:31.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.247+0000 7f47e8389640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:31.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.247+0000 7f47e8389640 10 monclient(hunting): picked mon.noname-b con 0x7f47e00b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:31.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.247+0000 7f47e8389640 10 monclient(hunting): picked mon.noname-a con 0x7f47e00b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:31.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.247+0000 7f47e8389640 10 monclient(hunting): picked mon.noname-c con 0x7f47e014be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:31.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.247+0000 7f47e8389640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e00af940 con 0x7f47e00b1730 2024-03-25T18:11:31.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.247+0000 7f47e8389640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e00b0a70 con 0x7f47e014be90 2024-03-25T18:11:31.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.247+0000 7f47e8389640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e00b0d50 con 0x7f47e00b4d60 2024-03-25T18:11:31.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.247+0000 7f47e8389640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:31.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.247+0000 7f47e8389640 10 monclient(hunting): authenticate will time out at 7046.500156s 2024-03-25T18:11:31.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.254+0000 7f47e6125640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f47e00b4d60 0x7f47e013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:39836/0 (socket says 172.21.15.37:39836) 2024-03-25T18:11:31.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.254+0000 7f47e6125640 1 -- 172.21.15.37:0/3235649470 learned_addr learned my addr 172.21.15.37:0/3235649470 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.267+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2845106952 0 0) 0x7f47e00b0d50 con 0x7f47e00b4d60 2024-03-25T18:11:31.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.267+0000 7f47e5123640 10 monclient(hunting): _negotiate 2024-03-25T18:11:31.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.267+0000 7f47e5123640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:31.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.267+0000 7f47e5123640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:31.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.267+0000 7f47e5123640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:11:31.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.267+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47cc003520 con 0x7f47e00b4d60 2024-03-25T18:11:31.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.278+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3711652238 0 0) 0x7f47e00b0a70 con 0x7f47e014be90 2024-03-25T18:11:31.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.278+0000 7f47e5123640 10 monclient(hunting): _negotiate 2024-03-25T18:11:31.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.278+0000 7f47e5123640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:31.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.278+0000 7f47e5123640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:31.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.278+0000 7f47e5123640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:11:31.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.278+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e00b0d50 con 0x7f47e014be90 2024-03-25T18:11:31.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.281+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3469257422 0 0) 0x7f47e00af940 con 0x7f47e00b1730 2024-03-25T18:11:31.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.281+0000 7f47e5123640 10 monclient(hunting): _negotiate 2024-03-25T18:11:31.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.281+0000 7f47e5123640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:31.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.281+0000 7f47e5123640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:31.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.281+0000 7f47e5123640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:11:31.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.281+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e00b0a70 con 0x7f47e00b1730 2024-03-25T18:11:31.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2454529646 0 0) 0x7f47cc003520 con 0x7f47e00b4d60 2024-03-25T18:11:31.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 >> v1:172.21.15.117:6789/0 conn(0x7f47e014be90 legacy=0x7f47e014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:31.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 >> v1:172.21.15.37:6789/0 conn(0x7f47e00b1730 legacy=0x7f47e00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:31.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 10 monclient: _finish_hunting 0 2024-03-25T18:11:31.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 1 monclient: found mon.noname-b 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e00b1030 con 0x7f47e00b4d60 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 10 monclient: _finish_auth 0 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 10 monclient: _check_auth_tickets 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f47dc001690 con 0x7f47e00b4d60 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e5123640 10 monclient: dump: 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.289 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:31.290 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:31.290 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:31.290 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:31.290 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:31.290 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:31.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e8389640 5 monclient: authenticate success, global_id 4385 2024-03-25T18:11:31.290 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.287+0000 7f47e8389640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:31.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.310+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47dc0017f0 con 0x7f47e00b4d60 2024-03-25T18:11:31.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.310+0000 7f47e5123640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:31.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.310+0000 7f47e5123640 1 -- 172.21.15.37:0/3235649470 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f47dc001c00 con 0x7f47e00b4d60 2024-03-25T18:11:31.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.310+0000 7f47e8389640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:31.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.310+0000 7f47e8389640 4 set_mon_vals no callback set 2024-03-25T18:11:31.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.310+0000 7f47e5123640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:31.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.310+0000 7f47e5123640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:31.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.310+0000 7f47e5123640 10 monclient: dump: 2024-03-25T18:11:31.311 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:31.312 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:31.312 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.312 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.312 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:31.312 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:31.312 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:31.312 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:31.312 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:31.312 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:31.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.310+0000 7f47e8389640 10 monclient: shutdown 2024-03-25T18:11:31.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.310+0000 7f47e8389640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:31.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.310+0000 7f47e8389640 1 -- 172.21.15.37:0/3235649470 >> v1:172.21.15.37:6790/0 conn(0x7f47e00b4d60 legacy=0x7f47e013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:31.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.311+0000 7f47e8389640 1 -- 172.21.15.37:0/3235649470 shutdown_connections 2024-03-25T18:11:31.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.311+0000 7f47e8389640 1 -- 172.21.15.37:0/3235649470 >> 172.21.15.37:0/3235649470 conn(0x7f47e00617a0 msgr2=0x7f47e0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:31.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.311+0000 7f47e8389640 1 -- 172.21.15.37:0/3235649470 shutdown_connections 2024-03-25T18:11:31.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.311+0000 7f47e8389640 1 -- 172.21.15.37:0/3235649470 wait complete. 2024-03-25T18:11:31.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.311+0000 7f47e8389640 10 monclient: build_initial_monmap 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.311+0000 7f47e8389640 10 monclient: monmap: 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 1 librados: starting msgr at 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 1 librados: starting objecter 2024-03-25T18:11:31.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 1 Processor -- start 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 1 -- start start 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 1 librados: setting wanted keys 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 1 librados: calling monclient init 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 10 monclient: init 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 10 monclient(hunting): picked mon.noname-a con 0x7f47e00b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 10 monclient(hunting): picked mon.noname-c con 0x7f47e00b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 10 monclient(hunting): picked mon.noname-b con 0x7f47e014be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e01442d0 con 0x7f47e00b1730 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e01445b0 con 0x7f47e014be90 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47e013e7a0 con 0x7f47e00b4d60 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:31.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.312+0000 7f47e8389640 10 monclient(hunting): authenticate will time out at 7046.565098s 2024-03-25T18:11:31.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.320+0000 7f47e5924640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f47e00b4d60 0x7f47e012e990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:39890/0 (socket says 172.21.15.37:39890) 2024-03-25T18:11:31.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.320+0000 7f47e5924640 1 -- 172.21.15.37:0/646923 learned_addr learned my addr 172.21.15.37:0/646923 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:31.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.322+0000 7f47d6ffd640 10 client.?.objecter ms_handle_connect 0x7f47e00b4d60 2024-03-25T18:11:31.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.322+0000 7f47d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:31.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.327+0000 7f47d6ffd640 10 client.?.objecter ms_handle_connect 0x7f47e00b1730 2024-03-25T18:11:31.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.327+0000 7f47d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:31.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.336+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922191416 0 0) 0x7f47e013e7a0 con 0x7f47e00b4d60 2024-03-25T18:11:31.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.336+0000 7f47d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:31.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.336+0000 7f47d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:31.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.336+0000 7f47d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:31.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.336+0000 7f47d6ffd640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:11:31.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.336+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c4004eb0 con 0x7f47e00b4d60 2024-03-25T18:11:31.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.338+0000 7f47d6ffd640 10 client.?.objecter ms_handle_connect 0x7f47e014be90 2024-03-25T18:11:31.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.338+0000 7f47d6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:31.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.340+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2970197704 0 0) 0x7f47e01442d0 con 0x7f47e00b1730 2024-03-25T18:11:31.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.340+0000 7f47d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:31.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.340+0000 7f47d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:31.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.340+0000 7f47d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:31.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.340+0000 7f47d6ffd640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:11:31.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.340+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e013e7a0 con 0x7f47e00b1730 2024-03-25T18:11:31.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.347+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450136777 0 0) 0x7f47e01445b0 con 0x7f47e014be90 2024-03-25T18:11:31.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.347+0000 7f47d6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:11:31.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.347+0000 7f47d6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:31.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.347+0000 7f47d6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:31.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.347+0000 7f47d6ffd640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:11:31.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.347+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47e01442d0 con 0x7f47e014be90 2024-03-25T18:11:31.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.376+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3629370349 0 0) 0x7f47c4004eb0 con 0x7f47e00b4d60 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.376+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47e01445b0 con 0x7f47e00b4d60 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.376+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f47d0001a10 con 0x7f47e00b4d60 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.376+0000 7f47d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.377+0000 7f47d6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.377+0000 7f47d6ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:31.378 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:31.379 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:31.379 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:31.379 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:31.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.377+0000 7f47d6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.379+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1960938367 0 0) 0x7f47e01442d0 con 0x7f47e014be90 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.379+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47c4004eb0 con 0x7f47e014be90 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.379+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f47d8001bd0 con 0x7f47e014be90 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.379+0000 7f47d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.379+0000 7f47d6ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.379+0000 7f47d6ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:31.381 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:31.382 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:31.382 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:31.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.385+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2879098897 0 0) 0x7f47e013e7a0 con 0x7f47e00b1730 2024-03-25T18:11:31.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.385+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47e01442d0 con 0x7f47e00b1730 2024-03-25T18:11:31.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.385+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f47dc001b00 con 0x7f47e00b1730 2024-03-25T18:11:31.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.385+0000 7f47d6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:31.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.385+0000 7f47d6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:31.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.385+0000 7f47d6ffd640 10 monclient(hunting): dump: 2024-03-25T18:11:31.387 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:31.387 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:31.387 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.387 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.387 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:31.387 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:31.387 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:31.387 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:31.387 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:31.387 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:31.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3983475627 0 0) 0x7f47e01445b0 con 0x7f47e00b4d60 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 >> v1:172.21.15.117:6789/0 conn(0x7f47e014be90 legacy=0x7f47e012f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 >> v1:172.21.15.37:6789/0 conn(0x7f47e00b1730 legacy=0x7f47e0144cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47d6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47d6ffd640 1 monclient: found mon.c 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47d6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47d6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e01453c0 con 0x7f47e00b4d60 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47d6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47d6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47d6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47e8389640 5 monclient: authenticate success, global_id 4388 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47e8389640 10 monclient: _renew_subs 2024-03-25T18:11:31.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47e8389640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:31.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47e8389640 1 -- 172.21.15.37:0/646923 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47e012f7b0 con 0x7f47e00b4d60 2024-03-25T18:11:31.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47e8389640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:31.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47e8389640 10 monclient: _renew_subs 2024-03-25T18:11:31.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47e8389640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:31.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47e8389640 1 -- 172.21.15.37:0/646923 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47e013e2b0 con 0x7f47e00b4d60 2024-03-25T18:11:31.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.400+0000 7f47e8389640 1 librados: init done 2024-03-25T18:11:31.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.401+0000 7f47e8389640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:31.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.401+0000 7f47e8389640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:31.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.401+0000 7f47e8389640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:31.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.401+0000 7f47e8389640 1 -- 172.21.15.37:0/646923 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47e00afbd0 con 0x7f47e00b4d60 2024-03-25T18:11:31.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.728+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47d0001c70 con 0x7f47e00b4d60 2024-03-25T18:11:31.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.728+0000 7f47d6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:31.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.728+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f47d0003a60 con 0x7f47e00b4d60 2024-03-25T18:11:31.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.728+0000 7f47d6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.728+0000 7f47d6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.728+0000 7f47d6ffd640 10 monclient: dump: 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:31.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.728+0000 7f47e4922640 4 set_mon_vals no callback set 2024-03-25T18:11:31.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.729+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f47d0003c80 con 0x7f47e00b4d60 2024-03-25T18:11:31.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.729+0000 7f47e6125640 10 monclient: get_auth_request con 0x7f47c406e0f0 auth_method 0 2024-03-25T18:11:31.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.730+0000 7f47d6ffd640 10 client.4388.objecter ms_handle_connect 0x7f47c406e0f0 2024-03-25T18:11:31.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.735+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f47d0054860 con 0x7f47e00b4d60 2024-03-25T18:11:31.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.735+0000 7f47d6ffd640 10 client.4388.objecter ms_dispatch 0x7f47e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:31.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.735+0000 7f47d6ffd640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:31.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.735+0000 7f47d6ffd640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:31.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.735+0000 7f47d6ffd640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-25T18:11:31.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.735+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47d0050370 con 0x7f47e00b4d60 2024-03-25T18:11:31.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.735+0000 7f47d6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:31.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.735+0000 7f47d6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:31.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.970+0000 7f47e8389640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:31.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.970+0000 7f47e8389640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:31.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.970+0000 7f47e8389640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:31.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:31.970+0000 7f47e8389640 1 -- 172.21.15.37:0/646923 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47e013e930 con 0x7f47e00b4d60 2024-03-25T18:11:32.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.014+0000 7f47d6ffd640 1 -- 172.21.15.37:0/646923 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f47d00522a0 con 0x7f47e00b4d60 2024-03-25T18:11:32.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.014+0000 7f47d6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:32.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.014+0000 7f47d6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:32.016 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:11:32.016 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:32.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.017+0000 7f47e8389640 10 librados: watch_flush enter 2024-03-25T18:11:32.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.017+0000 7f47e8389640 10 librados: watch_flush exit 2024-03-25T18:11:32.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.017+0000 7f47e8389640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-25T18:11:32.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.017+0000 7f47e8389640 10 client.4388.objecter successfully canceled tick 2024-03-25T18:11:32.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.017+0000 7f47e8389640 1 -- 172.21.15.37:0/646923 >> v1:172.21.15.117:6800/3703993921 conn(0x7f47c406e0f0 legacy=0x7f47c40705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:32.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.017+0000 7f47e8389640 10 monclient: shutdown 2024-03-25T18:11:32.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.017+0000 7f47e8389640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:32.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.017+0000 7f47e8389640 1 -- 172.21.15.37:0/646923 >> v1:172.21.15.37:6790/0 conn(0x7f47e00b4d60 legacy=0x7f47e012e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:32.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.017+0000 7f47e8389640 1 -- 172.21.15.37:0/646923 shutdown_connections 2024-03-25T18:11:32.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.018+0000 7f47e8389640 1 -- 172.21.15.37:0/646923 >> 172.21.15.37:0/646923 conn(0x7f47e00617a0 msgr2=0x7f47e00b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:32.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.018+0000 7f47e8389640 1 -- 172.21.15.37:0/646923 shutdown_connections 2024-03-25T18:11:32.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.018+0000 7f47e8389640 1 -- 172.21.15.37:0/646923 wait complete. 2024-03-25T18:11:32.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:32.018+0000 7f47e8389640 1 librados: shutdown 2024-03-25T18:11:32.031 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:38.837 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:38.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.956+0000 7fa63fd82640 10 monclient: get_monmap_and_config 2024-03-25T18:11:38.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.956+0000 7fa63fd82640 10 monclient: build_initial_monmap 2024-03-25T18:11:38.958 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.956+0000 7fa63fd82640 10 monclient: monmap: 2024-03-25T18:11:38.958 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:38.958 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:38.958 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:11:38.957634+0000 2024-03-25T18:11:38.958 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:11:38.957634+0000 2024-03-25T18:11:38.958 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:38.958 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:38.958 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:38.958 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:11:38.958 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:11:38.958 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.957+0000 7fa63fd82640 1 Processor -- start 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.957+0000 7fa63fd82640 1 -- start start 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.957+0000 7fa63fd82640 10 monclient: init 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.958+0000 7fa63fd82640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.958+0000 7fa63fd82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.958+0000 7fa63fd82640 10 monclient(hunting): picked mon.noname-a con 0x7fa6380b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.958+0000 7fa63fd82640 10 monclient(hunting): picked mon.noname-b con 0x7fa6380b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.958+0000 7fa63fd82640 10 monclient(hunting): picked mon.noname-c con 0x7fa63814be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.958+0000 7fa63fd82640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6380af940 con 0x7fa6380b4d60 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.958+0000 7fa63fd82640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6380b0a70 con 0x7fa63814be90 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.958+0000 7fa63fd82640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6380b0d50 con 0x7fa6380b1730 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.958+0000 7fa63fd82640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:38.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.958+0000 7fa63fd82640 10 monclient(hunting): authenticate will time out at 7054.210777s 2024-03-25T18:11:38.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.965+0000 7fa63e31f640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fa63814be90 0x7fa63814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:56088/0 (socket says 172.21.15.37:56088) 2024-03-25T18:11:38.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.965+0000 7fa63e31f640 1 -- 172.21.15.37:0/2368338717 learned_addr learned my addr 172.21.15.37:0/2368338717 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:38.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.981+0000 7fa63cb1c640 1 -- 172.21.15.37:0/2368338717 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3096319312 0 0) 0x7fa6380b0a70 con 0x7fa63814be90 2024-03-25T18:11:38.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.981+0000 7fa63cb1c640 10 monclient(hunting): _negotiate 2024-03-25T18:11:38.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.981+0000 7fa63cb1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:38.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.981+0000 7fa63cb1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:38.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.981+0000 7fa63cb1c640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:11:38.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.981+0000 7fa63cb1c640 1 -- 172.21.15.37:0/2368338717 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa624003520 con 0x7fa63814be90 2024-03-25T18:11:38.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.992+0000 7fa63cb1c640 1 -- 172.21.15.37:0/2368338717 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 54991937 0 0) 0x7fa6380b0d50 con 0x7fa6380b1730 2024-03-25T18:11:38.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.992+0000 7fa63cb1c640 10 monclient(hunting): _negotiate 2024-03-25T18:11:38.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.992+0000 7fa63cb1c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:38.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.992+0000 7fa63cb1c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:38.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.992+0000 7fa63cb1c640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:11:38.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.992+0000 7fa63cb1c640 1 -- 172.21.15.37:0/2368338717 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6380b0a70 con 0x7fa6380b1730 2024-03-25T18:11:38.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 1 -- 172.21.15.37:0/2368338717 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 365406314 0 0) 0x7fa624003520 con 0x7fa63814be90 2024-03-25T18:11:38.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 1 -- 172.21.15.37:0/2368338717 >> v1:172.21.15.37:6790/0 conn(0x7fa6380b1730 legacy=0x7fa6380b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:38.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 1 -- 172.21.15.37:0/2368338717 >> v1:172.21.15.37:6789/0 conn(0x7fa6380b4d60 legacy=0x7fa63813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:38.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 10 monclient: _finish_hunting 0 2024-03-25T18:11:38.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 1 monclient: found mon.noname-c 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 1 -- 172.21.15.37:0/2368338717 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6380b1030 con 0x7fa63814be90 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 10 monclient: _finish_auth 0 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 10 monclient: _check_auth_tickets 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 1 -- 172.21.15.37:0/2368338717 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa630001a10 con 0x7fa63814be90 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63fd82640 5 monclient: authenticate success, global_id 4396 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63fd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63cb1c640 10 monclient: dump: 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:38.996 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:38.997 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:38.997 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:38.997 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:38.997 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:38.997 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:38.997 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:38.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:38.994+0000 7fa63fd82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.014+0000 7fa63cb1c640 1 -- 172.21.15.37:0/2368338717 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa630001b70 con 0x7fa63814be90 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.014+0000 7fa63cb1c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.014+0000 7fa63cb1c640 1 -- 172.21.15.37:0/2368338717 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa630001f80 con 0x7fa63814be90 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.014+0000 7fa63cb1c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.014+0000 7fa63cb1c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.014+0000 7fa63cb1c640 10 monclient: dump: 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:39.015 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:39.016 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:39.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.014+0000 7fa63fd82640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:39.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.014+0000 7fa63fd82640 4 set_mon_vals no callback set 2024-03-25T18:11:39.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.015+0000 7fa63fd82640 10 monclient: shutdown 2024-03-25T18:11:39.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.015+0000 7fa63fd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:39.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.015+0000 7fa63fd82640 1 -- 172.21.15.37:0/2368338717 >> v1:172.21.15.117:6789/0 conn(0x7fa63814be90 legacy=0x7fa63814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:39.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.015+0000 7fa63fd82640 1 -- 172.21.15.37:0/2368338717 shutdown_connections 2024-03-25T18:11:39.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.015+0000 7fa63fd82640 1 -- 172.21.15.37:0/2368338717 >> 172.21.15.37:0/2368338717 conn(0x7fa6380617a0 msgr2=0x7fa638061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:39.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.015+0000 7fa63fd82640 1 -- 172.21.15.37:0/2368338717 shutdown_connections 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.015+0000 7fa63fd82640 1 -- 172.21.15.37:0/2368338717 wait complete. 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.016+0000 7fa63fd82640 10 monclient: build_initial_monmap 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.016+0000 7fa63fd82640 10 monclient: monmap: 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.016+0000 7fa63fd82640 1 librados: starting msgr at 2024-03-25T18:11:39.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.016+0000 7fa63fd82640 1 librados: starting objecter 2024-03-25T18:11:39.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.016+0000 7fa63fd82640 1 Processor -- start 2024-03-25T18:11:39.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.016+0000 7fa63fd82640 1 -- start start 2024-03-25T18:11:39.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.016+0000 7fa63fd82640 1 librados: setting wanted keys 2024-03-25T18:11:39.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.016+0000 7fa63fd82640 1 librados: calling monclient init 2024-03-25T18:11:39.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.016+0000 7fa63fd82640 10 monclient: init 2024-03-25T18:11:39.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.017+0000 7fa63fd82640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:39.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.017+0000 7fa63fd82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:11:39.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.017+0000 7fa63fd82640 10 monclient(hunting): picked mon.noname-b con 0x7fa6380b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:39.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.017+0000 7fa63fd82640 10 monclient(hunting): picked mon.noname-a con 0x7fa6380b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:39.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.017+0000 7fa63fd82640 10 monclient(hunting): picked mon.noname-c con 0x7fa63814be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:39.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.017+0000 7fa63fd82640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa638135ee0 con 0x7fa6380b4d60 2024-03-25T18:11:39.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.017+0000 7fa63fd82640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa638135b00 con 0x7fa6380b1730 2024-03-25T18:11:39.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.017+0000 7fa63fd82640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa638135c90 con 0x7fa63814be90 2024-03-25T18:11:39.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.017+0000 7fa63fd82640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:39.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.017+0000 7fa63fd82640 10 monclient(hunting): authenticate will time out at 7054.270028s 2024-03-25T18:11:39.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.025+0000 7fa63d31d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa6380b4d60 0x7fa6381d7730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41944/0 (socket says 172.21.15.37:41944) 2024-03-25T18:11:39.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.025+0000 7fa63d31d640 1 -- 172.21.15.37:0/2798891318 learned_addr learned my addr 172.21.15.37:0/2798891318 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:39.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.026+0000 7fa62e7fc640 10 client.?.objecter ms_handle_connect 0x7fa6380b4d60 2024-03-25T18:11:39.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.026+0000 7fa62e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:39.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.034+0000 7fa62e7fc640 10 client.?.objecter ms_handle_connect 0x7fa63814be90 2024-03-25T18:11:39.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.034+0000 7fa62e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:39.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.038+0000 7fa62e7fc640 10 client.?.objecter ms_handle_connect 0x7fa6380b1730 2024-03-25T18:11:39.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.038+0000 7fa62e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:39.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.053+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1464678981 0 0) 0x7fa638135c90 con 0x7fa63814be90 2024-03-25T18:11:39.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.053+0000 7fa62e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:39.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.053+0000 7fa62e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:39.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.053+0000 7fa62e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:39.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.053+0000 7fa62e7fc640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:11:39.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.053+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa61c004eb0 con 0x7fa63814be90 2024-03-25T18:11:39.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.058+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1394805733 0 0) 0x7fa638135b00 con 0x7fa6380b1730 2024-03-25T18:11:39.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.058+0000 7fa62e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:39.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.058+0000 7fa62e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:39.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.058+0000 7fa62e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:39.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.058+0000 7fa62e7fc640 10 monclient(hunting): my global_id is 4399 2024-03-25T18:11:39.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.058+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa638135c90 con 0x7fa6380b1730 2024-03-25T18:11:39.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.060+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369653147 0 0) 0x7fa638135ee0 con 0x7fa6380b4d60 2024-03-25T18:11:39.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.060+0000 7fa62e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:11:39.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.060+0000 7fa62e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:39.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.060+0000 7fa62e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:39.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.060+0000 7fa62e7fc640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:11:39.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.060+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa638135b00 con 0x7fa6380b4d60 2024-03-25T18:11:39.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2847049228 0 0) 0x7fa638135b00 con 0x7fa6380b4d60 2024-03-25T18:11:39.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa638135ee0 con 0x7fa6380b4d60 2024-03-25T18:11:39.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa628001a30 con 0x7fa6380b4d60 2024-03-25T18:11:39.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:39.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:39.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 10 monclient(hunting): dump: 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 378250227 0 0) 0x7fa638135c90 con 0x7fa6380b1730 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa638135b00 con 0x7fa6380b1730 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa634001820 con 0x7fa6380b1730 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:39.074 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:39.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.072+0000 7fa62e7fc640 10 monclient(hunting): dump: 2024-03-25T18:11:39.075 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:39.075 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:39.075 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:39.075 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:39.075 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:39.075 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:39.075 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:39.075 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:39.075 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:39.075 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:39.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.075+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 63285634 0 0) 0x7fa61c004eb0 con 0x7fa63814be90 2024-03-25T18:11:39.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.075+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa638135c90 con 0x7fa63814be90 2024-03-25T18:11:39.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.075+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa630001e80 con 0x7fa63814be90 2024-03-25T18:11:39.076 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.075+0000 7fa62e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.075+0000 7fa62e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.075+0000 7fa62e7fc640 10 monclient(hunting): dump: 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:39.077 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:39.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1698057684 0 0) 0x7fa638135b00 con 0x7fa6380b1730 2024-03-25T18:11:39.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 >> v1:172.21.15.37:6790/0 conn(0x7fa63814be90 legacy=0x7fa6381d7e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 >> v1:172.21.15.37:6789/0 conn(0x7fa6380b4d60 legacy=0x7fa6381d7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa62e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa62e7fc640 1 monclient: found mon.b 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa62e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa62e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa638136960 con 0x7fa6380b1730 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa62e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa62e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa62e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa63fd82640 5 monclient: authenticate success, global_id 4399 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa63fd82640 10 monclient: _renew_subs 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa63fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa63fd82640 1 -- 172.21.15.37:0/2798891318 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa638130470 con 0x7fa6380b1730 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa63fd82640 10 client.4399.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:39.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa63fd82640 10 monclient: _renew_subs 2024-03-25T18:11:39.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa63fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:39.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa63fd82640 1 -- 172.21.15.37:0/2798891318 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa63813e690 con 0x7fa6380b1730 2024-03-25T18:11:39.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.084+0000 7fa63fd82640 1 librados: init done 2024-03-25T18:11:39.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.085+0000 7fa63fd82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:39.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.085+0000 7fa63fd82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:39.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.085+0000 7fa63fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:39.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.085+0000 7fa63fd82640 1 -- 172.21.15.37:0/2798891318 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa638136b50 con 0x7fa6380b1730 2024-03-25T18:11:39.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.369+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa634001cf0 con 0x7fa6380b1730 2024-03-25T18:11:39.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.369+0000 7fa62e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.369+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa6340035f0 con 0x7fa6380b1730 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.369+0000 7fa62e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.369+0000 7fa62e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.369+0000 7fa62ffff640 4 set_mon_vals no callback set 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.369+0000 7fa62e7fc640 10 monclient: dump: 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:39.371 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:39.372 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:39.372 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:39.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.370+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fa6340195c0 con 0x7fa6380b1730 2024-03-25T18:11:39.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.371+0000 7fa63d31d640 10 monclient: get_auth_request con 0x7fa61c06e110 auth_method 0 2024-03-25T18:11:39.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.371+0000 7fa62e7fc640 10 client.4399.objecter ms_handle_connect 0x7fa61c06e110 2024-03-25T18:11:39.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.376+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fa6340532f0 con 0x7fa6380b1730 2024-03-25T18:11:39.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.376+0000 7fa62e7fc640 10 client.4399.objecter ms_dispatch 0x7fa638000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:39.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.376+0000 7fa62e7fc640 3 client.4399.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:39.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.376+0000 7fa62e7fc640 3 client.4399.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:39.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.376+0000 7fa62e7fc640 20 client.4399.objecter dump_active .. 0 homeless 2024-03-25T18:11:39.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.377+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa63404ec30 con 0x7fa6380b1730 2024-03-25T18:11:39.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.377+0000 7fa62e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:39.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.377+0000 7fa62e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:39.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.616+0000 7fa63fd82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:39.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.616+0000 7fa63fd82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:39.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.616+0000 7fa63fd82640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:11:39.617 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.616+0000 7fa63fd82640 1 -- 172.21.15.37:0/2798891318 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6380627b0 con 0x7fa6380b1730 2024-03-25T18:11:39.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.665+0000 7fa62e7fc640 1 -- 172.21.15.37:0/2798891318 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fa634050d30 con 0x7fa6380b1730 2024-03-25T18:11:39.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.665+0000 7fa62e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:39.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.665+0000 7fa62e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:39.667 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:11:39.667 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:39.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.667+0000 7fa63fd82640 10 librados: watch_flush enter 2024-03-25T18:11:39.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.667+0000 7fa63fd82640 10 librados: watch_flush exit 2024-03-25T18:11:39.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.667+0000 7fa63fd82640 20 client.4399.objecter shutdown clearing up homeless session... 2024-03-25T18:11:39.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.667+0000 7fa63fd82640 10 client.4399.objecter successfully canceled tick 2024-03-25T18:11:39.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.667+0000 7fa63fd82640 1 -- 172.21.15.37:0/2798891318 >> v1:172.21.15.117:6800/3703993921 conn(0x7fa61c06e110 legacy=0x7fa61c0705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:39.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.668+0000 7fa63fd82640 10 monclient: shutdown 2024-03-25T18:11:39.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.668+0000 7fa63fd82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:39.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.668+0000 7fa63fd82640 1 -- 172.21.15.37:0/2798891318 >> v1:172.21.15.117:6789/0 conn(0x7fa6380b1730 legacy=0x7fa63812fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:39.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.668+0000 7fa63fd82640 1 -- 172.21.15.37:0/2798891318 shutdown_connections 2024-03-25T18:11:39.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.668+0000 7fa63fd82640 1 -- 172.21.15.37:0/2798891318 >> 172.21.15.37:0/2798891318 conn(0x7fa6380617a0 msgr2=0x7fa6380b25c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:39.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.668+0000 7fa63fd82640 1 -- 172.21.15.37:0/2798891318 shutdown_connections 2024-03-25T18:11:39.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.668+0000 7fa63fd82640 1 -- 172.21.15.37:0/2798891318 wait complete. 2024-03-25T18:11:39.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:39.668+0000 7fa63fd82640 1 librados: shutdown 2024-03-25T18:11:39.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:46.487 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.603+0000 7f18970ae640 10 monclient: get_monmap_and_config 2024-03-25T18:11:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.603+0000 7f18970ae640 10 monclient: build_initial_monmap 2024-03-25T18:11:46.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.603+0000 7f18970ae640 10 monclient: monmap: 2024-03-25T18:11:46.604 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:46.605 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:46.605 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:11:46.604587+0000 2024-03-25T18:11:46.605 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:11:46.604587+0000 2024-03-25T18:11:46.605 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:46.605 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:46.605 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:46.605 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:11:46.605 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:11:46.605 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.604+0000 7f18970ae640 1 Processor -- start 2024-03-25T18:11:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.605+0000 7f18970ae640 1 -- start start 2024-03-25T18:11:46.606 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.605+0000 7f18970ae640 10 monclient: init 2024-03-25T18:11:46.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.606+0000 7f18970ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:46.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.606+0000 7f18970ae640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:11:46.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.606+0000 7f18970ae640 10 monclient(hunting): picked mon.noname-b con 0x7f18900a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:46.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.606+0000 7f18970ae640 10 monclient(hunting): picked mon.noname-a con 0x7f189013f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:46.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.606+0000 7f18970ae640 10 monclient(hunting): picked mon.noname-c con 0x7f18901430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:46.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.606+0000 7f18970ae640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1890142520 con 0x7f189013f710 2024-03-25T18:11:46.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.606+0000 7f18970ae640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18900b28a0 con 0x7f18901430c0 2024-03-25T18:11:46.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.606+0000 7f18970ae640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18900b2a30 con 0x7f18900a98f0 2024-03-25T18:11:46.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.606+0000 7f18970ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:46.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.606+0000 7f18970ae640 10 monclient(hunting): authenticate will time out at 7061.858549s 2024-03-25T18:11:46.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.617+0000 7f188ffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f189013f710 0x7f18901429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46450/0 (socket says 172.21.15.37:46450) 2024-03-25T18:11:46.618 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.617+0000 7f188ffff640 1 -- 172.21.15.37:0/1419379632 learned_addr learned my addr 172.21.15.37:0/1419379632 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:46.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.631+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3576869537 0 0) 0x7f1890142520 con 0x7f189013f710 2024-03-25T18:11:46.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.631+0000 7f188f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:46.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.631+0000 7f188f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:46.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.631+0000 7f188f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:46.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.631+0000 7f188f7fe640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:11:46.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.631+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1880003520 con 0x7f189013f710 2024-03-25T18:11:46.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.632+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1329999934 0 0) 0x7f18900b28a0 con 0x7f18901430c0 2024-03-25T18:11:46.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.632+0000 7f188f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:46.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.632+0000 7f188f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:46.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.632+0000 7f188f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:46.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.632+0000 7f188f7fe640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:11:46.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.632+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1890142520 con 0x7f18901430c0 2024-03-25T18:11:46.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.633+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2918300278 0 0) 0x7f18900b2a30 con 0x7f18900a98f0 2024-03-25T18:11:46.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.633+0000 7f188f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:46.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.633+0000 7f188f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:46.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.633+0000 7f188f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:46.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.633+0000 7f188f7fe640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:11:46.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.633+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18900b28a0 con 0x7f18900a98f0 2024-03-25T18:11:46.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2552938538 0 0) 0x7f1880003520 con 0x7f189013f710 2024-03-25T18:11:46.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 >> v1:172.21.15.37:6790/0 conn(0x7f18900a98f0 legacy=0x7f18900a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:46.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 >> v1:172.21.15.117:6789/0 conn(0x7f18901430c0 legacy=0x7f189014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:46.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:46.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:11:46.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:46.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:11:46.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18900b44c0 con 0x7f189013f710 2024-03-25T18:11:46.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:46.651 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f187c001690 con 0x7f189013f710 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f188f7fe640 10 monclient: dump: 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:46.652 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:46.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f18970ae640 5 monclient: authenticate success, global_id 4401 2024-03-25T18:11:46.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.650+0000 7f18970ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:46.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.660+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f187c0017f0 con 0x7f189013f710 2024-03-25T18:11:46.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.660+0000 7f188f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:46.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.660+0000 7f188f7fe640 1 -- 172.21.15.37:0/1419379632 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f187c001c00 con 0x7f189013f710 2024-03-25T18:11:46.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.660+0000 7f18970ae640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:46.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.660+0000 7f18970ae640 4 set_mon_vals no callback set 2024-03-25T18:11:46.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.660+0000 7f188f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:46.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.660+0000 7f188f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:46.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.660+0000 7f188f7fe640 10 monclient: dump: 2024-03-25T18:11:46.662 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:46.662 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:46.662 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:46.662 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:46.662 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:46.662 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:46.662 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:46.662 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:46.662 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:46.662 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:46.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.661+0000 7f18970ae640 10 monclient: shutdown 2024-03-25T18:11:46.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.661+0000 7f18970ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:46.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.661+0000 7f18970ae640 1 -- 172.21.15.37:0/1419379632 >> v1:172.21.15.37:6789/0 conn(0x7f189013f710 legacy=0x7f18901429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:46.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.661+0000 7f18970ae640 1 -- 172.21.15.37:0/1419379632 shutdown_connections 2024-03-25T18:11:46.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.661+0000 7f18970ae640 1 -- 172.21.15.37:0/1419379632 >> 172.21.15.37:0/1419379632 conn(0x7f18900b7ea0 msgr2=0x7f18900b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:46.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.661+0000 7f18970ae640 1 -- 172.21.15.37:0/1419379632 shutdown_connections 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.661+0000 7f18970ae640 1 -- 172.21.15.37:0/1419379632 wait complete. 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.661+0000 7f18970ae640 10 monclient: build_initial_monmap 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.661+0000 7f18970ae640 10 monclient: monmap: 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.662+0000 7f18970ae640 1 librados: starting msgr at 2024-03-25T18:11:46.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.662+0000 7f18970ae640 1 librados: starting objecter 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.662+0000 7f18970ae640 1 Processor -- start 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.662+0000 7f18970ae640 1 -- start start 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.662+0000 7f18970ae640 1 librados: setting wanted keys 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.662+0000 7f18970ae640 1 librados: calling monclient init 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.662+0000 7f18970ae640 10 monclient: init 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.663+0000 7f18970ae640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.663+0000 7f18970ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.663+0000 7f18970ae640 10 monclient(hunting): picked mon.noname-c con 0x7f18900a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.663+0000 7f18970ae640 10 monclient(hunting): picked mon.noname-b con 0x7f189013f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.663+0000 7f18970ae640 10 monclient(hunting): picked mon.noname-a con 0x7f18901430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.663+0000 7f18970ae640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18901d82d0 con 0x7f18901430c0 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.663+0000 7f18970ae640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18900abb90 con 0x7f189013f710 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.663+0000 7f18970ae640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18901d76a0 con 0x7f18900a98f0 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.663+0000 7f18970ae640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:46.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.663+0000 7f18970ae640 10 monclient(hunting): authenticate will time out at 7061.915477s 2024-03-25T18:11:46.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.675+0000 7f189564b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f18901430c0 0x7f1890134010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:46468/0 (socket says 172.21.15.37:46468) 2024-03-25T18:11:46.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.675+0000 7f189564b640 1 -- 172.21.15.37:0/3384333696 learned_addr learned my addr 172.21.15.37:0/3384333696 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:46.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.677+0000 7f188d7fa640 10 client.?.objecter ms_handle_connect 0x7f189013f710 2024-03-25T18:11:46.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.677+0000 7f188d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:46.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.679+0000 7f188d7fa640 10 client.?.objecter ms_handle_connect 0x7f18900a98f0 2024-03-25T18:11:46.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.679+0000 7f188d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:46.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.680+0000 7f188d7fa640 10 client.?.objecter ms_handle_connect 0x7f18901430c0 2024-03-25T18:11:46.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.680+0000 7f188d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:46.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.689+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 558553598 0 0) 0x7f18901d82d0 con 0x7f18901430c0 2024-03-25T18:11:46.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.689+0000 7f188d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:46.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.689+0000 7f188d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:46.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.689+0000 7f188d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:46.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.689+0000 7f188d7fa640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:11:46.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.689+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1874004eb0 con 0x7f18901430c0 2024-03-25T18:11:46.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.694+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3590110679 0 0) 0x7f18900abb90 con 0x7f189013f710 2024-03-25T18:11:46.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.694+0000 7f188d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:46.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.694+0000 7f188d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:46.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.694+0000 7f188d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:46.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.694+0000 7f188d7fa640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:11:46.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.694+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18901d82d0 con 0x7f189013f710 2024-03-25T18:11:46.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.701+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821969955 0 0) 0x7f18901d76a0 con 0x7f18900a98f0 2024-03-25T18:11:46.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.701+0000 7f188d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:46.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.701+0000 7f188d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:46.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.701+0000 7f188d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:46.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.701+0000 7f188d7fa640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:11:46.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.701+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18900abb90 con 0x7f18900a98f0 2024-03-25T18:11:46.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.709+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1820577649 0 0) 0x7f1874004eb0 con 0x7f18901430c0 2024-03-25T18:11:46.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.709+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18901d76a0 con 0x7f18901430c0 2024-03-25T18:11:46.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.709+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1888001c90 con 0x7f18901430c0 2024-03-25T18:11:46.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.709+0000 7f188d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:46.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.709+0000 7f188d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:11:46.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.709+0000 7f188d7fa640 10 monclient(hunting): dump: 2024-03-25T18:11:46.711 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:46.711 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:46.711 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:46.711 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:46.711 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:46.711 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:46.711 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:46.711 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:46.711 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:46.711 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:46.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.709+0000 7f188d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:46.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.721+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973590722 0 0) 0x7f18900abb90 con 0x7f18900a98f0 2024-03-25T18:11:46.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.721+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1874004eb0 con 0x7f18900a98f0 2024-03-25T18:11:46.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.721+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1884001870 con 0x7f18900a98f0 2024-03-25T18:11:46.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.721+0000 7f188d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:46.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.721+0000 7f188d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:46.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.721+0000 7f188d7fa640 10 monclient(hunting): dump: 2024-03-25T18:11:46.722 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:46.722 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:46.723 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:46.723 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:46.723 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:46.723 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:46.723 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:46.723 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:46.723 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:46.723 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:46.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 559023680 0 0) 0x7f18901d76a0 con 0x7f18901430c0 2024-03-25T18:11:46.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 >> v1:172.21.15.37:6790/0 conn(0x7f18900a98f0 legacy=0x7f18901385d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:46.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 >> v1:172.21.15.117:6789/0 conn(0x7f189013f710 legacy=0x7f18901d7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:46.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f188d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:11:46.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f188d7fa640 1 monclient: found mon.a 2024-03-25T18:11:46.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f188d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:46.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f188d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1890134720 con 0x7f18901430c0 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f188d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f188d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f188d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f18970ae640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.722+0000 7f18970ae640 10 monclient: _renew_subs 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.723+0000 7f18970ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.723+0000 7f18970ae640 1 -- 172.21.15.37:0/3384333696 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1890134970 con 0x7f18901430c0 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.723+0000 7f18970ae640 10 client.4404.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.723+0000 7f18970ae640 10 monclient: _renew_subs 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.723+0000 7f18970ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.723+0000 7f18970ae640 1 -- 172.21.15.37:0/3384333696 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1890137a10 con 0x7f18901430c0 2024-03-25T18:11:46.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.723+0000 7f18970ae640 1 librados: init done 2024-03-25T18:11:46.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.724+0000 7f18970ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:46.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.724+0000 7f18970ae640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:46.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.724+0000 7f18970ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:11:46.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:46.724+0000 7f18970ae640 1 -- 172.21.15.37:0/3384333696 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18901d7830 con 0x7f18901430c0 2024-03-25T18:11:47.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.042+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1888003990 con 0x7f18901430c0 2024-03-25T18:11:47.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.042+0000 7f188d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:47.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.042+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1888003ce0 con 0x7f18901430c0 2024-03-25T18:11:47.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.042+0000 7f188d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:47.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.042+0000 7f188d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:11:47.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.042+0000 7f188d7fa640 10 monclient: dump: 2024-03-25T18:11:47.043 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:47.043 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:47.043 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:47.044 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:47.044 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:47.044 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:47.044 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:47.044 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:47.044 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:47.044 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:47.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.042+0000 7f188e7fc640 4 set_mon_vals no callback set 2024-03-25T18:11:47.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.042+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f1888003f00 con 0x7f18901430c0 2024-03-25T18:11:47.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.043+0000 7f1894e4a640 10 monclient: get_auth_request con 0x7f187406df70 auth_method 0 2024-03-25T18:11:47.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.043+0000 7f188d7fa640 10 client.4404.objecter ms_handle_connect 0x7f187406df70 2024-03-25T18:11:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.050+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f1888054a80 con 0x7f18901430c0 2024-03-25T18:11:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.050+0000 7f188d7fa640 10 client.4404.objecter ms_dispatch 0x7f1890000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.050+0000 7f188d7fa640 3 client.4404.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.050+0000 7f188d7fa640 3 client.4404.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.050+0000 7f188d7fa640 20 client.4404.objecter dump_active .. 0 homeless 2024-03-25T18:11:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.050+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1888050590 con 0x7f18901430c0 2024-03-25T18:11:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.050+0000 7f188d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.050+0000 7f188d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:47.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.276+0000 7f18970ae640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:47.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.276+0000 7f18970ae640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:47.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.276+0000 7f18970ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:11:47.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.276+0000 7f18970ae640 1 -- 172.21.15.37:0/3384333696 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1890074cc0 con 0x7f18901430c0 2024-03-25T18:11:47.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.331+0000 7f188d7fa640 1 -- 172.21.15.37:0/3384333696 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f18880524c0 con 0x7f18901430c0 2024-03-25T18:11:47.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.331+0000 7f188d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:47.332 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.331+0000 7f188d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:47.332 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:11:47.332 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:47.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.333+0000 7f18970ae640 10 librados: watch_flush enter 2024-03-25T18:11:47.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.333+0000 7f18970ae640 10 librados: watch_flush exit 2024-03-25T18:11:47.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.333+0000 7f18970ae640 20 client.4404.objecter shutdown clearing up homeless session... 2024-03-25T18:11:47.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.333+0000 7f18970ae640 10 client.4404.objecter successfully canceled tick 2024-03-25T18:11:47.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.333+0000 7f18970ae640 1 -- 172.21.15.37:0/3384333696 >> v1:172.21.15.117:6800/3703993921 conn(0x7f187406df70 legacy=0x7f1874070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:47.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.333+0000 7f18970ae640 10 monclient: shutdown 2024-03-25T18:11:47.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.333+0000 7f18970ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:47.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.333+0000 7f18970ae640 1 -- 172.21.15.37:0/3384333696 >> v1:172.21.15.37:6789/0 conn(0x7f18901430c0 legacy=0x7f1890134010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:47.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.334+0000 7f18970ae640 1 -- 172.21.15.37:0/3384333696 shutdown_connections 2024-03-25T18:11:47.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.334+0000 7f18970ae640 1 -- 172.21.15.37:0/3384333696 >> 172.21.15.37:0/3384333696 conn(0x7f18900b7ea0 msgr2=0x7f1890074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:47.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.334+0000 7f18970ae640 1 -- 172.21.15.37:0/3384333696 shutdown_connections 2024-03-25T18:11:47.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.334+0000 7f18970ae640 1 -- 172.21.15.37:0/3384333696 wait complete. 2024-03-25T18:11:47.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:47.334+0000 7f18970ae640 1 librados: shutdown 2024-03-25T18:11:47.347 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:11:54.153 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.267+0000 7fb9ff025640 10 monclient: get_monmap_and_config 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.267+0000 7fb9ff025640 10 monclient: build_initial_monmap 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.267+0000 7fb9ff025640 10 monclient: monmap: 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:11:54.268237+0000 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:11:54.268237+0000 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:11:54.268 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:11:54.269 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:54.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.268+0000 7fb9ff025640 1 Processor -- start 2024-03-25T18:11:54.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.268+0000 7fb9ff025640 1 -- start start 2024-03-25T18:11:54.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.268+0000 7fb9ff025640 10 monclient: init 2024-03-25T18:11:54.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.269+0000 7fb9ff025640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:54.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.269+0000 7fb9ff025640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:11:54.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.269+0000 7fb9ff025640 10 monclient(hunting): picked mon.noname-b con 0x7fb9f813fc60 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:54.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.269+0000 7fb9ff025640 10 monclient(hunting): picked mon.noname-c con 0x7fb9f8070570 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:54.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.269+0000 7fb9ff025640 10 monclient(hunting): picked mon.noname-a con 0x7fb9f80727a0 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:54.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.269+0000 7fb9ff025640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f813c9f0 con 0x7fb9f80727a0 2024-03-25T18:11:54.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.269+0000 7fb9ff025640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f806fac0 con 0x7fb9f8070570 2024-03-25T18:11:54.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.269+0000 7fb9ff025640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f80715d0 con 0x7fb9f813fc60 2024-03-25T18:11:54.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.269+0000 7fb9ff025640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:54.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.269+0000 7fb9ff025640 10 monclient(hunting): authenticate will time out at 7069.521519s 2024-03-25T18:11:54.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.273+0000 7fb9fcdc1640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fb9f813fc60 0x7fb9f806f0a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:37162/0 (socket says 172.21.15.37:37162) 2024-03-25T18:11:54.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.273+0000 7fb9fcdc1640 1 -- 172.21.15.37:0/2785146874 learned_addr learned my addr 172.21.15.37:0/2785146874 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:54.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.286+0000 7fb9f77fe640 1 -- 172.21.15.37:0/2785146874 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1123195370 0 0) 0x7fb9f806fac0 con 0x7fb9f8070570 2024-03-25T18:11:54.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.286+0000 7fb9f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:11:54.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.286+0000 7fb9f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:54.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.286+0000 7fb9f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:54.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.286+0000 7fb9f77fe640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:11:54.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.286+0000 7fb9f77fe640 1 -- 172.21.15.37:0/2785146874 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9e8005730 con 0x7fb9f8070570 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 1 -- 172.21.15.37:0/2785146874 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1854005939 0 0) 0x7fb9e8005730 con 0x7fb9f8070570 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 1 -- 172.21.15.37:0/2785146874 >> v1:172.21.15.37:6790/0 conn(0x7fb9f813fc60 legacy=0x7fb9f806f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 1 -- 172.21.15.37:0/2785146874 >> v1:172.21.15.37:6789/0 conn(0x7fb9f80727a0 legacy=0x7fb9f814a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 1 monclient: found mon.noname-c 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 1 -- 172.21.15.37:0/2785146874 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f80718b0 con 0x7fb9f8070570 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 10 monclient: _finish_auth 0 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 1 -- 172.21.15.37:0/2785146874 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb9e4001690 con 0x7fb9f8070570 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:54.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.293+0000 7fb9f77fe640 10 monclient: dump: 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.294+0000 7fb9ff025640 5 monclient: authenticate success, global_id 4408 2024-03-25T18:11:54.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.294+0000 7fb9ff025640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.318+0000 7fb9f77fe640 1 -- 172.21.15.37:0/2785146874 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9e40017f0 con 0x7fb9f8070570 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9f77fe640 1 -- 172.21.15.37:0/2785146874 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb9e4001c40 con 0x7fb9f8070570 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9f77fe640 10 monclient: dump: 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:54.320 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:54.321 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:54.321 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:54.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9ff025640 10 monclient: get_monmap_and_config success 2024-03-25T18:11:54.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9ff025640 4 set_mon_vals no callback set 2024-03-25T18:11:54.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9ff025640 10 monclient: shutdown 2024-03-25T18:11:54.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9ff025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:54.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9ff025640 1 -- 172.21.15.37:0/2785146874 >> v1:172.21.15.117:6789/0 conn(0x7fb9f8070570 legacy=0x7fb9f8072090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9ff025640 1 -- 172.21.15.37:0/2785146874 shutdown_connections 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.319+0000 7fb9ff025640 1 -- 172.21.15.37:0/2785146874 >> 172.21.15.37:0/2785146874 conn(0x7fb9f80b4a80 msgr2=0x7fb9f80b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.320+0000 7fb9ff025640 1 -- 172.21.15.37:0/2785146874 shutdown_connections 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.320+0000 7fb9ff025640 1 -- 172.21.15.37:0/2785146874 wait complete. 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.320+0000 7fb9ff025640 10 monclient: build_initial_monmap 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.320+0000 7fb9ff025640 10 monclient: monmap: 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:11:54.323 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:11:54.324 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:54.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.321+0000 7fb9ff025640 1 librados: starting msgr at 2024-03-25T18:11:54.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.321+0000 7fb9ff025640 1 librados: starting objecter 2024-03-25T18:11:54.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.321+0000 7fb9ff025640 1 Processor -- start 2024-03-25T18:11:54.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.321+0000 7fb9ff025640 1 -- start start 2024-03-25T18:11:54.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.321+0000 7fb9ff025640 1 librados: setting wanted keys 2024-03-25T18:11:54.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.321+0000 7fb9ff025640 1 librados: calling monclient init 2024-03-25T18:11:54.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.321+0000 7fb9ff025640 10 monclient: init 2024-03-25T18:11:54.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.322+0000 7fb9ff025640 10 monclient: _reopen_session rank -1 2024-03-25T18:11:54.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.322+0000 7fb9ff025640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:11:54.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.322+0000 7fb9ff025640 10 monclient(hunting): picked mon.noname-a con 0x7fb9f8070570 addr v1:172.21.15.37:6789/0 2024-03-25T18:11:54.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.322+0000 7fb9ff025640 10 monclient(hunting): picked mon.noname-b con 0x7fb9f80727a0 addr v1:172.21.15.117:6789/0 2024-03-25T18:11:54.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.322+0000 7fb9ff025640 10 monclient(hunting): picked mon.noname-c con 0x7fb9f813fc60 addr v1:172.21.15.37:6790/0 2024-03-25T18:11:54.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.322+0000 7fb9ff025640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f80aabe0 con 0x7fb9f8070570 2024-03-25T18:11:54.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.322+0000 7fb9ff025640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f80aaec0 con 0x7fb9f80727a0 2024-03-25T18:11:54.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.322+0000 7fb9ff025640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9f8137f40 con 0x7fb9f813fc60 2024-03-25T18:11:54.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.322+0000 7fb9ff025640 10 monclient(hunting): _renew_subs 2024-03-25T18:11:54.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.322+0000 7fb9ff025640 10 monclient(hunting): authenticate will time out at 7069.574759s 2024-03-25T18:11:54.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.324+0000 7fb9fd5c2640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fb9f813fc60 0x7fb9f8140430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:37218/0 (socket says 172.21.15.37:37218) 2024-03-25T18:11:54.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.324+0000 7fb9fd5c2640 1 -- 172.21.15.37:0/3292535659 learned_addr learned my addr 172.21.15.37:0/3292535659 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:11:54.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.324+0000 7fb9f57fa640 10 client.?.objecter ms_handle_connect 0x7fb9f813fc60 2024-03-25T18:11:54.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.324+0000 7fb9f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:54.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.328+0000 7fb9f57fa640 10 client.?.objecter ms_handle_connect 0x7fb9f80727a0 2024-03-25T18:11:54.329 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.328+0000 7fb9f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:54.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.330+0000 7fb9f57fa640 10 client.?.objecter ms_handle_connect 0x7fb9f8070570 2024-03-25T18:11:54.331 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.330+0000 7fb9f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:11:54.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.334+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774913109 0 0) 0x7fb9f8137f40 con 0x7fb9f813fc60 2024-03-25T18:11:54.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.334+0000 7fb9f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:54.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.334+0000 7fb9f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:54.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.334+0000 7fb9f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:54.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.334+0000 7fb9f57fa640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:11:54.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.334+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc004ee0 con 0x7fb9f813fc60 2024-03-25T18:11:54.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1048839961 0 0) 0x7fb9f80aaec0 con 0x7fb9f80727a0 2024-03-25T18:11:54.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:54.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 10 monclient(hunting): my global_id is 4411 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9f8137f40 con 0x7fb9f80727a0 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3600480815 0 0) 0x7fb9dc004ee0 con 0x7fb9f813fc60 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9f80aaec0 con 0x7fb9f813fc60 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb9f0001c70 con 0x7fb9f813fc60 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 10 monclient(hunting): dump: 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:54.343 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:54.344 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:54.344 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:54.344 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:54.344 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:54.344 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:54.344 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:54.344 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:54.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.341+0000 7fb9f57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:11:54.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.346+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305981954 0 0) 0x7fb9f80aabe0 con 0x7fb9f8070570 2024-03-25T18:11:54.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.346+0000 7fb9f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:11:54.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.346+0000 7fb9f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:11:54.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.346+0000 7fb9f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:11:54.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.346+0000 7fb9f57fa640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:11:54.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.346+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9dc004ee0 con 0x7fb9f8070570 2024-03-25T18:11:54.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4201929646 0 0) 0x7fb9f80aaec0 con 0x7fb9f813fc60 2024-03-25T18:11:54.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 >> v1:172.21.15.117:6789/0 conn(0x7fb9f80727a0 legacy=0x7fb9f8137830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:54.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 >> v1:172.21.15.37:6789/0 conn(0x7fb9f8070570 legacy=0x7fb9f80a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:54.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9f57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:11:54.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9f57fa640 1 monclient: found mon.c 2024-03-25T18:11:54.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:11:54.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9f57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9f81380d0 con 0x7fb9f813fc60 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9f57fa640 10 monclient: _finish_auth 0 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9f57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9ff025640 5 monclient: authenticate success, global_id 4406 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9ff025640 10 monclient: _renew_subs 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9ff025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9ff025640 1 -- 172.21.15.37:0/3292535659 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9f8142e20 con 0x7fb9f813fc60 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9ff025640 10 client.4406.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9ff025640 10 monclient: _renew_subs 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9ff025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9ff025640 1 -- 172.21.15.37:0/3292535659 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9f8140b40 con 0x7fb9f813fc60 2024-03-25T18:11:54.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.356+0000 7fb9ff025640 1 librados: init done 2024-03-25T18:11:54.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.357+0000 7fb9ff025640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:54.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.357+0000 7fb9ff025640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:54.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.357+0000 7fb9ff025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:54.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.357+0000 7fb9ff025640 1 -- 172.21.15.37:0/3292535659 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9f80627b0 con 0x7fb9f813fc60 2024-03-25T18:11:54.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.673+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9f0003990 con 0x7fb9f813fc60 2024-03-25T18:11:54.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.673+0000 7fb9f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:11:54.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.673+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb9f0003ce0 con 0x7fb9f813fc60 2024-03-25T18:11:54.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.673+0000 7fb9f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:11:54.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.673+0000 7fb9f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:11:54.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.673+0000 7fb9f57fa640 10 monclient: dump: 2024-03-25T18:11:54.674 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:11:54.674 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:11:54.674 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:54.674 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:11:54.675 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:11:54.675 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:11:54.675 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:11:54.675 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:11:54.675 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:11:54.675 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:11:54.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.673+0000 7fb9f6ffd640 4 set_mon_vals no callback set 2024-03-25T18:11:54.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.673+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fb9f0003f00 con 0x7fb9f813fc60 2024-03-25T18:11:54.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.674+0000 7fb9fcdc1640 10 monclient: get_auth_request con 0x7fb9dc06de60 auth_method 0 2024-03-25T18:11:54.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.674+0000 7fb9f57fa640 10 client.4406.objecter ms_handle_connect 0x7fb9dc06de60 2024-03-25T18:11:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.687+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fb9f0054ad0 con 0x7fb9f813fc60 2024-03-25T18:11:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.687+0000 7fb9f57fa640 10 client.4406.objecter ms_dispatch 0x7fb9f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:11:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.687+0000 7fb9f57fa640 3 client.4406.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:11:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.687+0000 7fb9f57fa640 3 client.4406.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:11:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.687+0000 7fb9f57fa640 20 client.4406.objecter dump_active .. 0 homeless 2024-03-25T18:11:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.687+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb9f00505e0 con 0x7fb9f813fc60 2024-03-25T18:11:54.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.687+0000 7fb9f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:11:54.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.687+0000 7fb9f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:11:54.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.939+0000 7fb9ff025640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:54.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.939+0000 7fb9ff025640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:54.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.939+0000 7fb9ff025640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:11:54.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.939+0000 7fb9ff025640 1 -- 172.21.15.37:0/3292535659 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9f80af160 con 0x7fb9f813fc60 2024-03-25T18:11:54.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.965+0000 7fb9f57fa640 1 -- 172.21.15.37:0/3292535659 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fb9f0052510 con 0x7fb9f813fc60 2024-03-25T18:11:54.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.965+0000 7fb9f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:11:54.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.965+0000 7fb9f57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:11:54.966 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:11:54.966 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:11:54.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.967+0000 7fb9ff025640 10 librados: watch_flush enter 2024-03-25T18:11:54.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 10 librados: watch_flush exit 2024-03-25T18:11:54.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 20 client.4406.objecter shutdown clearing up homeless session... 2024-03-25T18:11:54.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 10 client.4406.objecter successfully canceled tick 2024-03-25T18:11:54.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 1 -- 172.21.15.37:0/3292535659 >> v1:172.21.15.117:6800/3703993921 conn(0x7fb9dc06de60 legacy=0x7fb9dc070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:54.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 10 monclient: shutdown 2024-03-25T18:11:54.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:11:54.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 1 -- 172.21.15.37:0/3292535659 >> v1:172.21.15.37:6790/0 conn(0x7fb9f813fc60 legacy=0x7fb9f8140430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:11:54.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 1 -- 172.21.15.37:0/3292535659 shutdown_connections 2024-03-25T18:11:54.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 1 -- 172.21.15.37:0/3292535659 >> 172.21.15.37:0/3292535659 conn(0x7fb9f80b4a80 msgr2=0x7fb9f806af80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:11:54.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 1 -- 172.21.15.37:0/3292535659 shutdown_connections 2024-03-25T18:11:54.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 1 -- 172.21.15.37:0/3292535659 wait complete. 2024-03-25T18:11:54.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:11:54.968+0000 7fb9ff025640 1 librados: shutdown 2024-03-25T18:11:54.981 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:01.788 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:01.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.902+0000 7f948e102640 10 monclient: get_monmap_and_config 2024-03-25T18:12:01.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.902+0000 7f948e102640 10 monclient: build_initial_monmap 2024-03-25T18:12:01.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.902+0000 7f948e102640 10 monclient: monmap: 2024-03-25T18:12:01.904 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:01.904 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:01.904 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:12:01.904031+0000 2024-03-25T18:12:01.905 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:12:01.904031+0000 2024-03-25T18:12:01.905 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:01.905 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:01.905 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:01.905 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:12:01.905 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:12:01.905 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:01.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.902+0000 7f948e102640 1 Processor -- start 2024-03-25T18:12:01.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 1 -- start start 2024-03-25T18:12:01.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 10 monclient: init 2024-03-25T18:12:01.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:01.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:01.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 10 monclient(hunting): picked mon.noname-b con 0x7f94880a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:01.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 10 monclient(hunting): picked mon.noname-c con 0x7f948813f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:01.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 10 monclient(hunting): picked mon.noname-a con 0x7f94881430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:01.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9488142520 con 0x7f94881430c0 2024-03-25T18:12:01.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94880b28a0 con 0x7f948813f710 2024-03-25T18:12:01.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94880b2a30 con 0x7f94880a98f0 2024-03-25T18:12:01.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:01.907 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.903+0000 7f948e102640 10 monclient(hunting): authenticate will time out at 7077.157079s 2024-03-25T18:12:01.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.912+0000 7f94877fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f94880a98f0 0x7f94880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:51544/0 (socket says 172.21.15.37:51544) 2024-03-25T18:12:01.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.912+0000 7f94877fe640 1 -- 172.21.15.37:0/3869524497 learned_addr learned my addr 172.21.15.37:0/3869524497 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:01.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.926+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 182988147 0 0) 0x7f94880b28a0 con 0x7f948813f710 2024-03-25T18:12:01.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.926+0000 7f94867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:01.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.926+0000 7f94867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:01.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.926+0000 7f94867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:01.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.926+0000 7f94867fc640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:12:01.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.926+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9470003520 con 0x7f948813f710 2024-03-25T18:12:01.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.927+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 53766451 0 0) 0x7f9488142520 con 0x7f94881430c0 2024-03-25T18:12:01.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.927+0000 7f94867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:01.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.927+0000 7f94867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:01.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.927+0000 7f94867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:01.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.927+0000 7f94867fc640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:12:01.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.927+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94880b28a0 con 0x7f94881430c0 2024-03-25T18:12:01.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.930+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547827335 0 0) 0x7f94880b2a30 con 0x7f94880a98f0 2024-03-25T18:12:01.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.930+0000 7f94867fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:01.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.930+0000 7f94867fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:01.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.930+0000 7f94867fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:01.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.930+0000 7f94867fc640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:12:01.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.930+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9488142520 con 0x7f94880a98f0 2024-03-25T18:12:01.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3675737103 0 0) 0x7f9470003520 con 0x7f948813f710 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 >> v1:172.21.15.37:6790/0 conn(0x7f94880a98f0 legacy=0x7f94880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 >> v1:172.21.15.37:6789/0 conn(0x7f94881430c0 legacy=0x7f948814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 1 monclient: found mon.noname-c 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94880b44c0 con 0x7f948813f710 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f947c001690 con 0x7f948813f710 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:01.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f94867fc640 10 monclient: dump: 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f948e102640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:12:01.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.933+0000 7f948e102640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:01.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.952+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f947c0017f0 con 0x7f948813f710 2024-03-25T18:12:01.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.952+0000 7f94867fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:01.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.952+0000 7f94867fc640 1 -- 172.21.15.37:0/3869524497 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f947c001c00 con 0x7f948813f710 2024-03-25T18:12:01.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.952+0000 7f94867fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:01.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.952+0000 7f94867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:01.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.952+0000 7f94867fc640 10 monclient: dump: 2024-03-25T18:12:01.955 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:01.955 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:01.955 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:01.955 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:01.955 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:01.955 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:01.955 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:01.955 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:01.955 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:01.955 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:01.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.952+0000 7f948e102640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:01.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.952+0000 7f948e102640 4 set_mon_vals no callback set 2024-03-25T18:12:01.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.953+0000 7f948e102640 10 monclient: shutdown 2024-03-25T18:12:01.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.953+0000 7f948e102640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:01.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.953+0000 7f948e102640 1 -- 172.21.15.37:0/3869524497 >> v1:172.21.15.117:6789/0 conn(0x7f948813f710 legacy=0x7f94881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:01.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.953+0000 7f948e102640 1 -- 172.21.15.37:0/3869524497 shutdown_connections 2024-03-25T18:12:01.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.953+0000 7f948e102640 1 -- 172.21.15.37:0/3869524497 >> 172.21.15.37:0/3869524497 conn(0x7f94880b7ea0 msgr2=0x7f94880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:01.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.953+0000 7f948e102640 1 -- 172.21.15.37:0/3869524497 shutdown_connections 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.953+0000 7f948e102640 1 -- 172.21.15.37:0/3869524497 wait complete. 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.954+0000 7f948e102640 10 monclient: build_initial_monmap 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.954+0000 7f948e102640 10 monclient: monmap: 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.954+0000 7f948e102640 1 librados: starting msgr at 2024-03-25T18:12:01.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.954+0000 7f948e102640 1 librados: starting objecter 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.954+0000 7f948e102640 1 Processor -- start 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.955+0000 7f948e102640 1 -- start start 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.955+0000 7f948e102640 1 librados: setting wanted keys 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.955+0000 7f948e102640 1 librados: calling monclient init 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.955+0000 7f948e102640 10 monclient: init 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.955+0000 7f948e102640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.955+0000 7f948e102640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.955+0000 7f948e102640 10 monclient(hunting): picked mon.noname-c con 0x7f94880a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.955+0000 7f948e102640 10 monclient(hunting): picked mon.noname-a con 0x7f948813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.955+0000 7f948e102640 10 monclient(hunting): picked mon.noname-b con 0x7f94881430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.955+0000 7f948e102640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94881d77a0 con 0x7f948813f710 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.956+0000 7f948e102640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9488138e40 con 0x7f94881430c0 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.956+0000 7f948e102640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94880aba30 con 0x7f94880a98f0 2024-03-25T18:12:01.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.956+0000 7f948e102640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:01.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.956+0000 7f948e102640 10 monclient(hunting): authenticate will time out at 7077.209500s 2024-03-25T18:12:01.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.959+0000 7f9487fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f94881430c0 0x7f9488134110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:33518/0 (socket says 172.21.15.37:33518) 2024-03-25T18:12:01.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.959+0000 7f9487fff640 1 -- 172.21.15.37:0/4264044271 learned_addr learned my addr 172.21.15.37:0/4264044271 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:01.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.960+0000 7f946ffff640 10 client.?.objecter ms_handle_connect 0x7f94881430c0 2024-03-25T18:12:01.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.960+0000 7f946ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:01.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.968+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 538748735 0 0) 0x7f9488138e40 con 0x7f94881430c0 2024-03-25T18:12:01.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.968+0000 7f946ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:01.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.968+0000 7f946ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:01.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.968+0000 7f946ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:01.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.968+0000 7f946ffff640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:12:01.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.968+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9474004eb0 con 0x7f94881430c0 2024-03-25T18:12:01.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.970+0000 7f946ffff640 10 client.?.objecter ms_handle_connect 0x7f94880a98f0 2024-03-25T18:12:01.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.970+0000 7f946ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:01.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.976+0000 7f946ffff640 10 client.?.objecter ms_handle_connect 0x7f948813f710 2024-03-25T18:12:01.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.976+0000 7f946ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:01.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.978+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3856323661 0 0) 0x7f94880aba30 con 0x7f94880a98f0 2024-03-25T18:12:01.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.978+0000 7f946ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:01.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.978+0000 7f946ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:01.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.978+0000 7f946ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:01.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.978+0000 7f946ffff640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:12:01.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.978+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9488138e40 con 0x7f94880a98f0 2024-03-25T18:12:01.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.981+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3230263710 0 0) 0x7f9474004eb0 con 0x7f94881430c0 2024-03-25T18:12:01.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.981+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94880aba30 con 0x7f94881430c0 2024-03-25T18:12:01.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.981+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9480001c90 con 0x7f94881430c0 2024-03-25T18:12:01.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.981+0000 7f946ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:01.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.981+0000 7f946ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.981+0000 7f946ffff640 10 monclient(hunting): dump: 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:01.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.981+0000 7f946ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:01.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.988+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2428765777 0 0) 0x7f94881d77a0 con 0x7f948813f710 2024-03-25T18:12:01.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.988+0000 7f946ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:01.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.988+0000 7f946ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:01.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.988+0000 7f946ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:01.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.988+0000 7f946ffff640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:12:01.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.988+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9474004eb0 con 0x7f948813f710 2024-03-25T18:12:01.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.989+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3341806063 0 0) 0x7f9488138e40 con 0x7f94880a98f0 2024-03-25T18:12:01.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.989+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94881d77a0 con 0x7f94880a98f0 2024-03-25T18:12:01.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.989+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9478001870 con 0x7f94880a98f0 2024-03-25T18:12:01.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.989+0000 7f946ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:01.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.989+0000 7f946ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:01.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:01.989+0000 7f946ffff640 10 monclient(hunting): dump: 2024-03-25T18:12:01.991 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:01.991 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:01.991 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:01.991 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:01.992 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:01.992 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:01.992 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:01.992 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:01.992 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:01.992 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:02.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2679984654 0 0) 0x7f94881d77a0 con 0x7f94880a98f0 2024-03-25T18:12:02.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 >> v1:172.21.15.117:6789/0 conn(0x7f94881430c0 legacy=0x7f9488134110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 >> v1:172.21.15.37:6789/0 conn(0x7f948813f710 legacy=0x7f94881d7c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f946ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f946ffff640 1 monclient: found mon.c 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f946ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f946ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94881d8340 con 0x7f94880a98f0 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f946ffff640 10 monclient: _finish_auth 0 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f946ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f946ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f948e102640 5 monclient: authenticate success, global_id 4412 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f948e102640 10 monclient: _renew_subs 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f948e102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f948e102640 1 -- 172.21.15.37:0/4264044271 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9488134820 con 0x7f94880a98f0 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f948e102640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:02.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f948e102640 10 monclient: _renew_subs 2024-03-25T18:12:02.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f948e102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:12:02.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f948e102640 1 -- 172.21.15.37:0/4264044271 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94881380c0 con 0x7f94880a98f0 2024-03-25T18:12:02.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.000+0000 7f948e102640 1 librados: init done 2024-03-25T18:12:02.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.001+0000 7f948e102640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:02.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.001+0000 7f948e102640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:02.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.001+0000 7f948e102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:12:02.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.001+0000 7f948e102640 1 -- 172.21.15.37:0/4264044271 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94880627b0 con 0x7f94880a98f0 2024-03-25T18:12:02.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.332+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9478001cc0 con 0x7f94880a98f0 2024-03-25T18:12:02.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.332+0000 7f946ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:02.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.332+0000 7f9485ffb640 4 set_mon_vals no callback set 2024-03-25T18:12:02.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.333+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9478004a40 con 0x7f94880a98f0 2024-03-25T18:12:02.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.333+0000 7f946ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:02.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.333+0000 7f946ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.333+0000 7f946ffff640 10 monclient: dump: 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:02.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.333+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f9478004c60 con 0x7f94880a98f0 2024-03-25T18:12:02.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.334+0000 7f9486ffd640 10 monclient: get_auth_request con 0x7f947406df90 auth_method 0 2024-03-25T18:12:02.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.334+0000 7f946ffff640 10 client.4412.objecter ms_handle_connect 0x7f947406df90 2024-03-25T18:12:02.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.340+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f94780548b0 con 0x7f94880a98f0 2024-03-25T18:12:02.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.340+0000 7f946ffff640 10 client.4412.objecter ms_dispatch 0x7f9488000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:02.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.340+0000 7f946ffff640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:02.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.340+0000 7f946ffff640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:02.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.340+0000 7f946ffff640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-25T18:12:02.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.340+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f94780503c0 con 0x7f94880a98f0 2024-03-25T18:12:02.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.340+0000 7f946ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:02.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.340+0000 7f946ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:02.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.593+0000 7f948e102640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:02.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.594+0000 7f948e102640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:02.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.594+0000 7f948e102640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:12:02.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.594+0000 7f948e102640 1 -- 172.21.15.37:0/4264044271 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f948814e8b0 con 0x7f94880a98f0 2024-03-25T18:12:02.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.768+0000 7f946ffff640 1 -- 172.21.15.37:0/4264044271 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f94780522f0 con 0x7f94880a98f0 2024-03-25T18:12:02.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.768+0000 7f946ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:02.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.768+0000 7f946ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:02.771 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:12:02.771 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:02.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 10 librados: watch_flush enter 2024-03-25T18:12:02.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 10 librados: watch_flush exit 2024-03-25T18:12:02.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-25T18:12:02.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 10 client.4412.objecter successfully canceled tick 2024-03-25T18:12:02.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 1 -- 172.21.15.37:0/4264044271 >> v1:172.21.15.117:6800/3703993921 conn(0x7f947406df90 legacy=0x7f9474070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 10 monclient: shutdown 2024-03-25T18:12:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 1 -- 172.21.15.37:0/4264044271 >> v1:172.21.15.37:6790/0 conn(0x7f94880a98f0 legacy=0x7f9488138730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 1 -- 172.21.15.37:0/4264044271 shutdown_connections 2024-03-25T18:12:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 1 -- 172.21.15.37:0/4264044271 >> 172.21.15.37:0/4264044271 conn(0x7f94880b7ea0 msgr2=0x7f9488074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 1 -- 172.21.15.37:0/4264044271 shutdown_connections 2024-03-25T18:12:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 1 -- 172.21.15.37:0/4264044271 wait complete. 2024-03-25T18:12:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:02.771+0000 7f948e102640 1 librados: shutdown 2024-03-25T18:12:02.786 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:09.592 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:09.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.709+0000 7fb94ba01640 10 monclient: get_monmap_and_config 2024-03-25T18:12:09.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.709+0000 7fb94ba01640 10 monclient: build_initial_monmap 2024-03-25T18:12:09.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.709+0000 7fb94ba01640 10 monclient: monmap: 2024-03-25T18:12:09.711 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:09.711 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:09.711 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:12:09.710592+0000 2024-03-25T18:12:09.711 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:12:09.710592+0000 2024-03-25T18:12:09.711 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:09.711 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:09.711 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:09.711 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:12:09.711 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:12:09.711 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:09.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.710+0000 7fb94ba01640 1 Processor -- start 2024-03-25T18:12:09.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.710+0000 7fb94ba01640 1 -- start start 2024-03-25T18:12:09.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.710+0000 7fb94ba01640 10 monclient: init 2024-03-25T18:12:09.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.711+0000 7fb94ba01640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:09.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.711+0000 7fb94ba01640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:09.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.711+0000 7fb94ba01640 10 monclient(hunting): picked mon.noname-c con 0x7fb9440b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:09.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.711+0000 7fb94ba01640 10 monclient(hunting): picked mon.noname-b con 0x7fb9440b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:09.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.711+0000 7fb94ba01640 10 monclient(hunting): picked mon.noname-a con 0x7fb94414be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:09.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.711+0000 7fb94ba01640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9440af940 con 0x7fb94414be90 2024-03-25T18:12:09.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.711+0000 7fb94ba01640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9440b0a70 con 0x7fb9440b4d60 2024-03-25T18:12:09.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.711+0000 7fb94ba01640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9440b0d50 con 0x7fb9440b1730 2024-03-25T18:12:09.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.711+0000 7fb94ba01640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:09.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.711+0000 7fb94ba01640 10 monclient(hunting): authenticate will time out at 7084.963714s 2024-03-25T18:12:09.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.719+0000 7fb94979d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fb9440b4d60 0x7fb94413f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:52062/0 (socket says 172.21.15.37:52062) 2024-03-25T18:12:09.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.719+0000 7fb94979d640 1 -- 172.21.15.37:0/3145435093 learned_addr learned my addr 172.21.15.37:0/3145435093 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:09.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.738+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1356568804 0 0) 0x7fb9440b0a70 con 0x7fb9440b4d60 2024-03-25T18:12:09.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.738+0000 7fb93bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:09.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.738+0000 7fb93bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:09.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.738+0000 7fb93bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:09.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.738+0000 7fb93bfff640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:12:09.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.738+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb930003520 con 0x7fb9440b4d60 2024-03-25T18:12:09.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.744+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1980036311 0 0) 0x7fb9440b0d50 con 0x7fb9440b1730 2024-03-25T18:12:09.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.744+0000 7fb93bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:09.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.744+0000 7fb93bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:09.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.744+0000 7fb93bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:09.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.744+0000 7fb93bfff640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:12:09.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.744+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9440b0a70 con 0x7fb9440b1730 2024-03-25T18:12:09.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.750+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3598951023 0 0) 0x7fb9440af940 con 0x7fb94414be90 2024-03-25T18:12:09.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.750+0000 7fb93bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:09.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.750+0000 7fb93bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:09.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.750+0000 7fb93bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:09.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.750+0000 7fb93bfff640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:12:09.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.750+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb9440b0d50 con 0x7fb94414be90 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 104645788 0 0) 0x7fb930003520 con 0x7fb9440b4d60 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 >> v1:172.21.15.37:6790/0 conn(0x7fb9440b1730 legacy=0x7fb9440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 >> v1:172.21.15.37:6789/0 conn(0x7fb94414be90 legacy=0x7fb94414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 1 monclient: found mon.noname-c 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9440b1030 con 0x7fb9440b4d60 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 10 monclient: _finish_auth 0 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:09.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb940001690 con 0x7fb9440b4d60 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb93bfff640 10 monclient: dump: 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb94ba01640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:12:09.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.751+0000 7fb94ba01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:09.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.762+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb9400017f0 con 0x7fb9440b4d60 2024-03-25T18:12:09.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.762+0000 7fb93bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:09.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.762+0000 7fb93bfff640 1 -- 172.21.15.37:0/3145435093 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb940001c00 con 0x7fb9440b4d60 2024-03-25T18:12:09.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.762+0000 7fb94ba01640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:09.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.762+0000 7fb94ba01640 4 set_mon_vals no callback set 2024-03-25T18:12:09.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.762+0000 7fb93bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:09.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.762+0000 7fb93bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:09.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.762+0000 7fb93bfff640 10 monclient: dump: 2024-03-25T18:12:09.763 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:09.764 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:09.764 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:09.764 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:09.764 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:09.764 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:09.764 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:09.764 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:09.764 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:09.764 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:09.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.763+0000 7fb94ba01640 10 monclient: shutdown 2024-03-25T18:12:09.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.763+0000 7fb94ba01640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:09.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.763+0000 7fb94ba01640 1 -- 172.21.15.37:0/3145435093 >> v1:172.21.15.117:6789/0 conn(0x7fb9440b4d60 legacy=0x7fb94413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:09.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.763+0000 7fb94ba01640 1 -- 172.21.15.37:0/3145435093 shutdown_connections 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.763+0000 7fb94ba01640 1 -- 172.21.15.37:0/3145435093 >> 172.21.15.37:0/3145435093 conn(0x7fb9440617a0 msgr2=0x7fb944061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.763+0000 7fb94ba01640 1 -- 172.21.15.37:0/3145435093 shutdown_connections 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.763+0000 7fb94ba01640 1 -- 172.21.15.37:0/3145435093 wait complete. 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.763+0000 7fb94ba01640 10 monclient: build_initial_monmap 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.763+0000 7fb94ba01640 10 monclient: monmap: 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:12:09.767 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.764+0000 7fb94ba01640 1 librados: starting msgr at 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.764+0000 7fb94ba01640 1 librados: starting objecter 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.764+0000 7fb94ba01640 1 Processor -- start 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.764+0000 7fb94ba01640 1 -- start start 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.764+0000 7fb94ba01640 1 librados: setting wanted keys 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.764+0000 7fb94ba01640 1 librados: calling monclient init 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.764+0000 7fb94ba01640 10 monclient: init 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.765+0000 7fb94ba01640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.765+0000 7fb94ba01640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.765+0000 7fb94ba01640 10 monclient(hunting): picked mon.noname-b con 0x7fb9440b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.765+0000 7fb94ba01640 10 monclient(hunting): picked mon.noname-c con 0x7fb9440b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.765+0000 7fb94ba01640 10 monclient(hunting): picked mon.noname-a con 0x7fb94414be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.765+0000 7fb94ba01640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb944140ce0 con 0x7fb94414be90 2024-03-25T18:12:09.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.765+0000 7fb94ba01640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9441d7a00 con 0x7fb9440b1730 2024-03-25T18:12:09.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.765+0000 7fb94ba01640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb9441d85a0 con 0x7fb9440b4d60 2024-03-25T18:12:09.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.765+0000 7fb94ba01640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:09.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.765+0000 7fb94ba01640 10 monclient(hunting): authenticate will time out at 7085.017421s 2024-03-25T18:12:09.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.767+0000 7fb948f9c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fb9440b4d60 0x7fb94413b5b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:34378/0 (socket says 172.21.15.37:34378) 2024-03-25T18:12:09.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.767+0000 7fb948f9c640 1 -- 172.21.15.37:0/1351255120 learned_addr learned my addr 172.21.15.37:0/1351255120 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:09.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.768+0000 7fb939ffb640 10 client.?.objecter ms_handle_connect 0x7fb9440b4d60 2024-03-25T18:12:09.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.768+0000 7fb939ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:09.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.772+0000 7fb939ffb640 10 client.?.objecter ms_handle_connect 0x7fb94414be90 2024-03-25T18:12:09.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.772+0000 7fb939ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:09.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.773+0000 7fb939ffb640 10 client.?.objecter ms_handle_connect 0x7fb9440b1730 2024-03-25T18:12:09.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.773+0000 7fb939ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:09.825 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:04:45.170 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.834+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1144192297 0 0) 0x7fb9440b0d50 con 0x7fb94414be90 2024-03-25T18:12:09.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.834+0000 7fb939ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:12:09.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.834+0000 7fb939ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:09.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.834+0000 7fb939ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:09.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.834+0000 7fb939ffb640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:12:09.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.834+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb928004eb0 con 0x7fb94414be90 2024-03-25T18:12:09.839 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:04:45.184 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:09.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.860+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 102744968 0 0) 0x7fb928004eb0 con 0x7fb94414be90 2024-03-25T18:12:09.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.860+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb9440b0d50 con 0x7fb94414be90 2024-03-25T18:12:09.861 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.860+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb93c001a30 con 0x7fb94414be90 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.860+0000 7fb939ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.860+0000 7fb939ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.860+0000 7fb939ffb640 10 monclient(hunting): dump: 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:09.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.860+0000 7fb939ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:09.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1569389096 0 0) 0x7fb9440b0d50 con 0x7fb94414be90 2024-03-25T18:12:09.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 >> v1:172.21.15.37:6790/0 conn(0x7fb9440b4d60 legacy=0x7fb94413b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:09.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 >> v1:172.21.15.117:6789/0 conn(0x7fb9440b1730 legacy=0x7fb9441d7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:09.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb939ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb939ffb640 1 monclient: found mon.a 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb939ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb939ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9441d7690 con 0x7fb94414be90 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb939ffb640 10 monclient: _finish_auth 0 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb939ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb939ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb94ba01640 5 monclient: authenticate success, global_id 4422 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb94ba01640 10 monclient: _renew_subs 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb94ba01640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb94ba01640 1 -- 172.21.15.37:0/1351255120 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb94413bcc0 con 0x7fb94414be90 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb94ba01640 10 client.4422.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:09.868 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb94ba01640 10 monclient: _renew_subs 2024-03-25T18:12:09.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb94ba01640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:09.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb94ba01640 1 -- 172.21.15.37:0/1351255120 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9441405f0 con 0x7fb94414be90 2024-03-25T18:12:09.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.866+0000 7fb94ba01640 1 librados: init done 2024-03-25T18:12:09.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.867+0000 7fb94ba01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:09.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.867+0000 7fb94ba01640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:09.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.867+0000 7fb94ba01640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:09.869 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:09.867+0000 7fb94ba01640 1 -- 172.21.15.37:0/1351255120 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9441d7820 con 0x7fb94414be90 2024-03-25T18:12:09.899 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:04:45.212 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:09.906 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:04:45.219 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:09.915 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:04:45.260 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:09.925 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:04:45.270 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:09.949 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:04:45.262 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:09.965 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:04:45.310 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:10.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.174+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb93c001f00 con 0x7fb94414be90 2024-03-25T18:12:10.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.174+0000 7fb939ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:10.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.174+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fb93c003b30 con 0x7fb94414be90 2024-03-25T18:12:10.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.174+0000 7fb939ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:10.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.174+0000 7fb939ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.174+0000 7fb939ffb640 10 monclient: dump: 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:10.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.174+0000 7fb93b7fe640 4 set_mon_vals no callback set 2024-03-25T18:12:10.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.175+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fb93c003d50 con 0x7fb94414be90 2024-03-25T18:12:10.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.175+0000 7fb94979d640 10 monclient: get_auth_request con 0x7fb92806dd00 auth_method 0 2024-03-25T18:12:10.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.176+0000 7fb939ffb640 10 client.4422.objecter ms_handle_connect 0x7fb92806dd00 2024-03-25T18:12:10.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.183+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fb93c053670 con 0x7fb94414be90 2024-03-25T18:12:10.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.183+0000 7fb939ffb640 10 client.4422.objecter ms_dispatch 0x7fb944000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:10.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.183+0000 7fb939ffb640 3 client.4422.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:10.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.183+0000 7fb939ffb640 3 client.4422.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:10.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.183+0000 7fb939ffb640 20 client.4422.objecter dump_active .. 0 homeless 2024-03-25T18:12:10.184 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:04:45.529 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:10.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.183+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb93c0b08b0 con 0x7fb94414be90 2024-03-25T18:12:10.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.183+0000 7fb939ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:10.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.183+0000 7fb939ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:10.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.435+0000 7fb94ba01640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:10.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.435+0000 7fb94ba01640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:10.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.435+0000 7fb94ba01640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:10.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.435+0000 7fb94ba01640 1 -- 172.21.15.37:0/1351255120 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb9440627b0 con 0x7fb94414be90 2024-03-25T18:12:10.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.465+0000 7fb939ffb640 1 -- 172.21.15.37:0/1351255120 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fb93c04f180 con 0x7fb94414be90 2024-03-25T18:12:10.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.465+0000 7fb939ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:10.466 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.465+0000 7fb939ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:10.467 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:12:10.467 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 10 librados: watch_flush enter 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 10 librados: watch_flush exit 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 20 client.4422.objecter shutdown clearing up homeless session... 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 10 client.4422.objecter successfully canceled tick 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 1 -- 172.21.15.37:0/1351255120 >> v1:172.21.15.117:6800/3703993921 conn(0x7fb92806dd00 legacy=0x7fb9280701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 10 monclient: shutdown 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 1 -- 172.21.15.37:0/1351255120 >> v1:172.21.15.37:6789/0 conn(0x7fb94414be90 legacy=0x7fb944142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 1 -- 172.21.15.37:0/1351255120 shutdown_connections 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 1 -- 172.21.15.37:0/1351255120 >> 172.21.15.37:0/1351255120 conn(0x7fb9440617a0 msgr2=0x7fb944061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 1 -- 172.21.15.37:0/1351255120 shutdown_connections 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 1 -- 172.21.15.37:0/1351255120 wait complete. 2024-03-25T18:12:10.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:10.468+0000 7fb94ba01640 1 librados: shutdown 2024-03-25T18:12:10.482 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:17.287 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:17.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.404+0000 7f159ea93640 10 monclient: get_monmap_and_config 2024-03-25T18:12:17.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.404+0000 7f159ea93640 10 monclient: build_initial_monmap 2024-03-25T18:12:17.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.404+0000 7f159ea93640 10 monclient: monmap: 2024-03-25T18:12:17.406 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:17.406 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:17.406 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:12:17.405508+0000 2024-03-25T18:12:17.406 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:12:17.405508+0000 2024-03-25T18:12:17.406 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:17.406 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:17.406 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:17.406 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:12:17.406 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:12:17.406 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:17.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.405+0000 7f159ea93640 1 Processor -- start 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.405+0000 7f159ea93640 1 -- start start 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.405+0000 7f159ea93640 10 monclient: init 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.405+0000 7f159ea93640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.406+0000 7f159ea93640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.406+0000 7f159ea93640 10 monclient(hunting): picked mon.noname-b con 0x7f15980a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.406+0000 7f159ea93640 10 monclient(hunting): picked mon.noname-c con 0x7f159813f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.406+0000 7f159ea93640 10 monclient(hunting): picked mon.noname-a con 0x7f15981430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.406+0000 7f159ea93640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1598142520 con 0x7f15981430c0 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.406+0000 7f159ea93640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15980b28a0 con 0x7f159813f710 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.406+0000 7f159ea93640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15980b2a30 con 0x7f15980a98f0 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.406+0000 7f159ea93640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:17.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.406+0000 7f159ea93640 10 monclient(hunting): authenticate will time out at 7092.658457s 2024-03-25T18:12:17.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.415+0000 7f159d030640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f15981430c0 0x7f159814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:37764/0 (socket says 172.21.15.37:37764) 2024-03-25T18:12:17.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.415+0000 7f159d030640 1 -- 172.21.15.37:0/2830041658 learned_addr learned my addr 172.21.15.37:0/2830041658 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:17.435 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 1 -- 172.21.15.37:0/2830041658 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2001859927 0 0) 0x7f1598142520 con 0x7f15981430c0 2024-03-25T18:12:17.435 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:17.435 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:17.435 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:17.435 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:12:17.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 1 -- 172.21.15.37:0/2830041658 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1588003520 con 0x7f15981430c0 2024-03-25T18:12:17.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 1 -- 172.21.15.37:0/2830041658 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3449299218 0 0) 0x7f15980b2a30 con 0x7f15980a98f0 2024-03-25T18:12:17.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:17.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:17.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:17.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 10 monclient(hunting): my global_id is 4421 2024-03-25T18:12:17.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.434+0000 7f15977fe640 1 -- 172.21.15.37:0/2830041658 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1598142520 con 0x7f15980a98f0 2024-03-25T18:12:17.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 1 -- 172.21.15.37:0/2830041658 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2232171756 0 0) 0x7f1598142520 con 0x7f15980a98f0 2024-03-25T18:12:17.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 1 -- 172.21.15.37:0/2830041658 >> v1:172.21.15.117:6789/0 conn(0x7f159813f710 legacy=0x7f15981429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:17.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 1 -- 172.21.15.37:0/2830041658 >> v1:172.21.15.37:6789/0 conn(0x7f15981430c0 legacy=0x7f159814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:17.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:17.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 1 monclient: found mon.noname-b 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 1 -- 172.21.15.37:0/2830041658 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15980b44c0 con 0x7f15980a98f0 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 1 -- 172.21.15.37:0/2830041658 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f158c001670 con 0x7f15980a98f0 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f159ea93640 5 monclient: authenticate success, global_id 4421 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f159ea93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f15977fe640 10 monclient: dump: 2024-03-25T18:12:17.446 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:17.447 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:17.447 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.447 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.447 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:17.447 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:17.447 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:17.447 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:17.447 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:17.447 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:17.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.444+0000 7f159ea93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:17.470 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.469+0000 7f15977fe640 1 -- 172.21.15.37:0/2830041658 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f158c0017d0 con 0x7f15980a98f0 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.469+0000 7f15977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.469+0000 7f15977fe640 1 -- 172.21.15.37:0/2830041658 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f158c001ba0 con 0x7f15980a98f0 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.469+0000 7f15977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.469+0000 7f15977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.469+0000 7f15977fe640 10 monclient: dump: 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:17.471 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:17.472 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:17.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.469+0000 7f159ea93640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:17.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.469+0000 7f159ea93640 4 set_mon_vals no callback set 2024-03-25T18:12:17.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.470+0000 7f159ea93640 10 monclient: shutdown 2024-03-25T18:12:17.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.470+0000 7f159ea93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:17.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.470+0000 7f159ea93640 1 -- 172.21.15.37:0/2830041658 >> v1:172.21.15.37:6790/0 conn(0x7f15980a98f0 legacy=0x7f15980a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:17.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.470+0000 7f159ea93640 1 -- 172.21.15.37:0/2830041658 shutdown_connections 2024-03-25T18:12:17.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.470+0000 7f159ea93640 1 -- 172.21.15.37:0/2830041658 >> 172.21.15.37:0/2830041658 conn(0x7f15980b7ea0 msgr2=0x7f15980b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:17.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.470+0000 7f159ea93640 1 -- 172.21.15.37:0/2830041658 shutdown_connections 2024-03-25T18:12:17.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.470+0000 7f159ea93640 1 -- 172.21.15.37:0/2830041658 wait complete. 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.471+0000 7f159ea93640 10 monclient: build_initial_monmap 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.471+0000 7f159ea93640 10 monclient: monmap: 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.472+0000 7f159ea93640 1 librados: starting msgr at 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.472+0000 7f159ea93640 1 librados: starting objecter 2024-03-25T18:12:17.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.472+0000 7f159ea93640 1 Processor -- start 2024-03-25T18:12:17.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.472+0000 7f159ea93640 1 -- start start 2024-03-25T18:12:17.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.472+0000 7f159ea93640 1 librados: setting wanted keys 2024-03-25T18:12:17.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.472+0000 7f159ea93640 1 librados: calling monclient init 2024-03-25T18:12:17.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.472+0000 7f159ea93640 10 monclient: init 2024-03-25T18:12:17.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.473+0000 7f159ea93640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:17.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.473+0000 7f159ea93640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:12:17.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.473+0000 7f159ea93640 10 monclient(hunting): picked mon.noname-c con 0x7f15980a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:17.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.473+0000 7f159ea93640 10 monclient(hunting): picked mon.noname-a con 0x7f159813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:17.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.473+0000 7f159ea93640 10 monclient(hunting): picked mon.noname-b con 0x7f15981430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:17.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.473+0000 7f159ea93640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159814e910 con 0x7f159813f710 2024-03-25T18:12:17.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.473+0000 7f159ea93640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15980abb90 con 0x7f15981430c0 2024-03-25T18:12:17.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.473+0000 7f159ea93640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f159814e5c0 con 0x7f15980a98f0 2024-03-25T18:12:17.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.473+0000 7f159ea93640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:17.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.473+0000 7f159ea93640 10 monclient(hunting): authenticate will time out at 7092.725673s 2024-03-25T18:12:17.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.481+0000 7f1597fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f159813f710 0x7f15981d7870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:37810/0 (socket says 172.21.15.37:37810) 2024-03-25T18:12:17.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.481+0000 7f1597fff640 1 -- 172.21.15.37:0/2782523134 learned_addr learned my addr 172.21.15.37:0/2782523134 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:17.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.486+0000 7f15957fa640 10 client.?.objecter ms_handle_connect 0x7f159813f710 2024-03-25T18:12:17.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.486+0000 7f15957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:17.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.487+0000 7f15957fa640 10 client.?.objecter ms_handle_connect 0x7f15980a98f0 2024-03-25T18:12:17.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.487+0000 7f15957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:17.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.492+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2486224751 0 0) 0x7f159814e910 con 0x7f159813f710 2024-03-25T18:12:17.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.492+0000 7f15957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:17.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.492+0000 7f15957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:17.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.492+0000 7f15957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:17.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.492+0000 7f15957fa640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:12:17.493 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.492+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f157c004eb0 con 0x7f159813f710 2024-03-25T18:12:17.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.493+0000 7f15957fa640 10 client.?.objecter ms_handle_connect 0x7f15981430c0 2024-03-25T18:12:17.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.493+0000 7f15957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:17.505 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923085671 0 0) 0x7f159814e5c0 con 0x7f15980a98f0 2024-03-25T18:12:17.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:17.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:17.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:17.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:12:17.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f159814e910 con 0x7f15980a98f0 2024-03-25T18:12:17.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3581089547 0 0) 0x7f15980abb90 con 0x7f15981430c0 2024-03-25T18:12:17.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:17.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:17.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:17.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:12:17.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.504+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f159814e5c0 con 0x7f15981430c0 2024-03-25T18:12:17.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.519+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3980575348 0 0) 0x7f159814e5c0 con 0x7f15981430c0 2024-03-25T18:12:17.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.519+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15980abb90 con 0x7f15981430c0 2024-03-25T18:12:17.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.519+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1590001c10 con 0x7f15981430c0 2024-03-25T18:12:17.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.519+0000 7f15957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:17.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.519+0000 7f15957fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:17.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.519+0000 7f15957fa640 10 monclient(hunting): dump: 2024-03-25T18:12:17.521 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:17.521 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:17.521 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.521 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.521 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:17.521 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:17.521 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:17.521 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:17.521 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:17.521 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:17.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.519+0000 7f15957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:17.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.524+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1925390784 0 0) 0x7f159814e910 con 0x7f15980a98f0 2024-03-25T18:12:17.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.524+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f159814e5c0 con 0x7f15980a98f0 2024-03-25T18:12:17.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.524+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f158c001ae0 con 0x7f15980a98f0 2024-03-25T18:12:17.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.524+0000 7f15957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:17.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.524+0000 7f15957fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:17.525 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.524+0000 7f15957fa640 10 monclient(hunting): dump: 2024-03-25T18:12:17.526 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:17.526 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:17.526 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.526 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.526 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:17.526 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:17.526 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:17.526 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:17.526 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:17.526 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.527+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 550735799 0 0) 0x7f157c004eb0 con 0x7f159813f710 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.528+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f159814e910 con 0x7f159813f710 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.528+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f15840019d0 con 0x7f159813f710 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.528+0000 7f15957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.528+0000 7f15957fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.528+0000 7f15957fa640 10 monclient(hunting): dump: 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:17.529 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:17.530 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:17.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4110330094 0 0) 0x7f159814e910 con 0x7f159813f710 2024-03-25T18:12:17.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 >> v1:172.21.15.37:6790/0 conn(0x7f15980a98f0 legacy=0x7f15981383c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:17.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 >> v1:172.21.15.117:6789/0 conn(0x7f15981430c0 legacy=0x7f15981d7f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:17.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f15957fa640 10 monclient: _finish_hunting 0 2024-03-25T18:12:17.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f15957fa640 1 monclient: found mon.a 2024-03-25T18:12:17.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f15957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:17.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f15957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1598138ad0 con 0x7f159813f710 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f15957fa640 10 monclient: _finish_auth 0 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f15957fa640 10 monclient: _check_auth_tickets 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f15957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f159ea93640 5 monclient: authenticate success, global_id 4428 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f159ea93640 10 monclient: _renew_subs 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f159ea93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f159ea93640 1 -- 172.21.15.37:0/2782523134 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1598138d70 con 0x7f159813f710 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f159ea93640 10 client.4428.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f159ea93640 10 monclient: _renew_subs 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f159ea93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f159ea93640 1 -- 172.21.15.37:0/2782523134 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15981e32a0 con 0x7f159813f710 2024-03-25T18:12:17.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.541+0000 7f159ea93640 1 librados: init done 2024-03-25T18:12:17.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.542+0000 7f159ea93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:17.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.542+0000 7f159ea93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:17.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.542+0000 7f159ea93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:17.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.542+0000 7f159ea93640 1 -- 172.21.15.37:0/2782523134 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15980627b0 con 0x7f159813f710 2024-03-25T18:12:17.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.861+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1584003610 con 0x7f159813f710 2024-03-25T18:12:17.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.861+0000 7f15957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:17.862 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.861+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f15840039e0 con 0x7f159813f710 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.861+0000 7f15957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.861+0000 7f15957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.861+0000 7f15957fa640 10 monclient: dump: 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:17.863 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.861+0000 7f1596ffd640 4 set_mon_vals no callback set 2024-03-25T18:12:17.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.862+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f1584003c00 con 0x7f159813f710 2024-03-25T18:12:17.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.863+0000 7f159c82f640 10 monclient: get_auth_request con 0x7f157c06e1f0 auth_method 0 2024-03-25T18:12:17.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.863+0000 7f15957fa640 10 client.4428.objecter ms_handle_connect 0x7f157c06e1f0 2024-03-25T18:12:17.870 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.869+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f15840547b0 con 0x7f159813f710 2024-03-25T18:12:17.870 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.869+0000 7f15957fa640 10 client.4428.objecter ms_dispatch 0x7f1598000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:17.870 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.869+0000 7f15957fa640 3 client.4428.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:17.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.869+0000 7f15957fa640 3 client.4428.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:17.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.869+0000 7f15957fa640 20 client.4428.objecter dump_active .. 0 homeless 2024-03-25T18:12:17.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.869+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f15840502c0 con 0x7f159813f710 2024-03-25T18:12:17.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.869+0000 7f15957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:17.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:17.869+0000 7f15957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:18.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.103+0000 7f159ea93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:18.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.103+0000 7f159ea93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:18.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.103+0000 7f159ea93640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:18.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.103+0000 7f159ea93640 1 -- 172.21.15.37:0/2782523134 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1598141c20 con 0x7f159813f710 2024-03-25T18:12:18.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.131+0000 7f15957fa640 1 -- 172.21.15.37:0/2782523134 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f15840521f0 con 0x7f159813f710 2024-03-25T18:12:18.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.131+0000 7f15957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:18.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.131+0000 7f15957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:18.132 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:12:18.132 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:18.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.133+0000 7f159ea93640 10 librados: watch_flush enter 2024-03-25T18:12:18.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.133+0000 7f159ea93640 10 librados: watch_flush exit 2024-03-25T18:12:18.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.133+0000 7f159ea93640 20 client.4428.objecter shutdown clearing up homeless session... 2024-03-25T18:12:18.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.133+0000 7f159ea93640 10 client.4428.objecter successfully canceled tick 2024-03-25T18:12:18.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.133+0000 7f159ea93640 1 -- 172.21.15.37:0/2782523134 >> v1:172.21.15.117:6800/3703993921 conn(0x7f157c06e1f0 legacy=0x7f157c0706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:18.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.133+0000 7f159ea93640 10 monclient: shutdown 2024-03-25T18:12:18.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.133+0000 7f159ea93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:18.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.133+0000 7f159ea93640 1 -- 172.21.15.37:0/2782523134 >> v1:172.21.15.37:6789/0 conn(0x7f159813f710 legacy=0x7f15981d7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:18.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.134+0000 7f159ea93640 1 -- 172.21.15.37:0/2782523134 shutdown_connections 2024-03-25T18:12:18.135 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.134+0000 7f159ea93640 1 -- 172.21.15.37:0/2782523134 >> 172.21.15.37:0/2782523134 conn(0x7f15980b7ea0 msgr2=0x7f1598074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:18.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.134+0000 7f159ea93640 1 -- 172.21.15.37:0/2782523134 shutdown_connections 2024-03-25T18:12:18.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.134+0000 7f159ea93640 1 -- 172.21.15.37:0/2782523134 wait complete. 2024-03-25T18:12:18.136 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:18.134+0000 7f159ea93640 1 librados: shutdown 2024-03-25T18:12:18.146 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:24.952 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:25.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.066+0000 7fc58e3c8640 10 monclient: get_monmap_and_config 2024-03-25T18:12:25.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.066+0000 7fc58e3c8640 10 monclient: build_initial_monmap 2024-03-25T18:12:25.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.066+0000 7fc58e3c8640 10 monclient: monmap: 2024-03-25T18:12:25.067 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:25.067 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:25.067 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:12:25.067080+0000 2024-03-25T18:12:25.067 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:12:25.067080+0000 2024-03-25T18:12:25.067 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:25.067 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:25.067 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:25.067 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:12:25.068 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:12:25.068 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:25.068 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 1 Processor -- start 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 1 -- start start 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 10 monclient: init 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc5880a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc58813f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc5881430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc588142520 con 0x7fc5881430c0 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880b28a0 con 0x7fc58813f710 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5880b2a30 con 0x7fc5880a98f0 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:25.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.067+0000 7fc58e3c8640 10 monclient(hunting): authenticate will time out at 7100.320268s 2024-03-25T18:12:25.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.071+0000 7fc587fff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fc5880a98f0 0x7fc5880a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43958/0 (socket says 172.21.15.37:43958) 2024-03-25T18:12:25.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.071+0000 7fc587fff640 1 -- 172.21.15.37:0/612568376 learned_addr learned my addr 172.21.15.37:0/612568376 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:25.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.080+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2213610104 0 0) 0x7fc5880b2a30 con 0x7fc5880a98f0 2024-03-25T18:12:25.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.080+0000 7fc586ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:25.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.080+0000 7fc586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:25.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.080+0000 7fc586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:25.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.080+0000 7fc586ffd640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:12:25.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.080+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc570003520 con 0x7fc5880a98f0 2024-03-25T18:12:25.096 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 908501187 0 0) 0x7fc5880b28a0 con 0x7fc58813f710 2024-03-25T18:12:25.096 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:25.096 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:25.096 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:25.096 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:12:25.096 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5880b2a30 con 0x7fc58813f710 2024-03-25T18:12:25.096 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 236916590 0 0) 0x7fc588142520 con 0x7fc5881430c0 2024-03-25T18:12:25.096 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:25.096 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:25.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:25.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:12:25.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.095+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5880b28a0 con 0x7fc5881430c0 2024-03-25T18:12:25.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 27180086 0 0) 0x7fc570003520 con 0x7fc5880a98f0 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 >> v1:172.21.15.117:6789/0 conn(0x7fc58813f710 legacy=0x7fc5881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 >> v1:172.21.15.37:6789/0 conn(0x7fc5881430c0 legacy=0x7fc58814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 1 monclient: found mon.noname-b 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5880b44c0 con 0x7fc5880a98f0 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 10 monclient: _finish_auth 0 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fc578001690 con 0x7fc5880a98f0 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc586ffd640 10 monclient: dump: 2024-03-25T18:12:25.099 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:25.100 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:25.100 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:25.100 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:25.100 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:25.100 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:25.100 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:25.100 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:25.100 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:25.100 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:25.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc58e3c8640 5 monclient: authenticate success, global_id 4427 2024-03-25T18:12:25.100 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.097+0000 7fc58e3c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:25.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.117+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5780017f0 con 0x7fc5880a98f0 2024-03-25T18:12:25.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.117+0000 7fc586ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:25.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.117+0000 7fc586ffd640 1 -- 172.21.15.37:0/612568376 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fc578001c00 con 0x7fc5880a98f0 2024-03-25T18:12:25.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.117+0000 7fc58e3c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:25.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.117+0000 7fc58e3c8640 4 set_mon_vals no callback set 2024-03-25T18:12:25.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.117+0000 7fc586ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:25.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.117+0000 7fc586ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:25.118 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.117+0000 7fc586ffd640 10 monclient: dump: 2024-03-25T18:12:25.118 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:25.118 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:25.118 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:25.119 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:25.119 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:25.119 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:25.119 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:25.119 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:25.119 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:25.119 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.118+0000 7fc58e3c8640 10 monclient: shutdown 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.118+0000 7fc58e3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.118+0000 7fc58e3c8640 1 -- 172.21.15.37:0/612568376 >> v1:172.21.15.37:6790/0 conn(0x7fc5880a98f0 legacy=0x7fc5880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.118+0000 7fc58e3c8640 1 -- 172.21.15.37:0/612568376 shutdown_connections 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.118+0000 7fc58e3c8640 1 -- 172.21.15.37:0/612568376 >> 172.21.15.37:0/612568376 conn(0x7fc5880b7ea0 msgr2=0x7fc5880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.118+0000 7fc58e3c8640 1 -- 172.21.15.37:0/612568376 shutdown_connections 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.118+0000 7fc58e3c8640 1 -- 172.21.15.37:0/612568376 wait complete. 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.118+0000 7fc58e3c8640 10 monclient: build_initial_monmap 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.119+0000 7fc58e3c8640 10 monclient: monmap: 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:25.120 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:12:25.121 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:12:25.121 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:25.121 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:12:25.121 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:25.121 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:12:25.121 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:12:25.121 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:25.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.119+0000 7fc58e3c8640 1 librados: starting msgr at 2024-03-25T18:12:25.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.119+0000 7fc58e3c8640 1 librados: starting objecter 2024-03-25T18:12:25.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.119+0000 7fc58e3c8640 1 Processor -- start 2024-03-25T18:12:25.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.119+0000 7fc58e3c8640 1 -- start start 2024-03-25T18:12:25.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.119+0000 7fc58e3c8640 1 librados: setting wanted keys 2024-03-25T18:12:25.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.119+0000 7fc58e3c8640 1 librados: calling monclient init 2024-03-25T18:12:25.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.119+0000 7fc58e3c8640 10 monclient: init 2024-03-25T18:12:25.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.120+0000 7fc58e3c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:25.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.120+0000 7fc58e3c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:25.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.120+0000 7fc58e3c8640 10 monclient(hunting): picked mon.noname-c con 0x7fc5880a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:25.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.120+0000 7fc58e3c8640 10 monclient(hunting): picked mon.noname-b con 0x7fc58813f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:25.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.120+0000 7fc58e3c8640 10 monclient(hunting): picked mon.noname-a con 0x7fc5881430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:25.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.120+0000 7fc58e3c8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc58813be70 con 0x7fc5881430c0 2024-03-25T18:12:25.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.120+0000 7fc58e3c8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc588131660 con 0x7fc58813f710 2024-03-25T18:12:25.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.120+0000 7fc58e3c8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5881317f0 con 0x7fc5880a98f0 2024-03-25T18:12:25.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.120+0000 7fc58e3c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:25.123 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.120+0000 7fc58e3c8640 10 monclient(hunting): authenticate will time out at 7100.372787s 2024-03-25T18:12:25.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.128+0000 7fc58c965640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fc5881430c0 0x7fc5881e2fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:37384/0 (socket says 172.21.15.37:37384) 2024-03-25T18:12:25.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.128+0000 7fc58c965640 1 -- 172.21.15.37:0/3441895775 learned_addr learned my addr 172.21.15.37:0/3441895775 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:25.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.130+0000 7fc584ff9640 10 client.?.objecter ms_handle_connect 0x7fc5881430c0 2024-03-25T18:12:25.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.130+0000 7fc584ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:25.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.133+0000 7fc584ff9640 10 client.?.objecter ms_handle_connect 0x7fc58813f710 2024-03-25T18:12:25.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.133+0000 7fc584ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:25.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.143+0000 7fc584ff9640 10 client.?.objecter ms_handle_connect 0x7fc5880a98f0 2024-03-25T18:12:25.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.143+0000 7fc584ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:25.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165040699 0 0) 0x7fc58813be70 con 0x7fc5881430c0 2024-03-25T18:12:25.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:25.145 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:25.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:25.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:12:25.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc574004eb0 con 0x7fc5881430c0 2024-03-25T18:12:25.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472272899 0 0) 0x7fc588131660 con 0x7fc58813f710 2024-03-25T18:12:25.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:25.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:25.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:25.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:12:25.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.144+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc58813be70 con 0x7fc58813f710 2024-03-25T18:12:25.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.157+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2357092134 0 0) 0x7fc58813be70 con 0x7fc58813f710 2024-03-25T18:12:25.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.157+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc588131660 con 0x7fc58813f710 2024-03-25T18:12:25.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.157+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fc57c001970 con 0x7fc58813f710 2024-03-25T18:12:25.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.157+0000 7fc584ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:25.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.157+0000 7fc584ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:25.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.157+0000 7fc584ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:25.158 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:25.158 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:25.159 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:25.159 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:25.159 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:25.159 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:25.159 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:25.159 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:25.159 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:25.159 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:25.159 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.157+0000 7fc584ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:25.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.164+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1552806352 0 0) 0x7fc5881317f0 con 0x7fc5880a98f0 2024-03-25T18:12:25.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.164+0000 7fc584ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:25.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.164+0000 7fc584ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:25.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.164+0000 7fc584ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:25.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.164+0000 7fc584ff9640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:12:25.166 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.164+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc58813be70 con 0x7fc5880a98f0 2024-03-25T18:12:25.171 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.170+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 19684823 0 0) 0x7fc574004eb0 con 0x7fc5881430c0 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.170+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5881317f0 con 0x7fc5881430c0 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.170+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fc580001c70 con 0x7fc5881430c0 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.170+0000 7fc584ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.170+0000 7fc584ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.170+0000 7fc584ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:25.172 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:25.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4078741324 0 0) 0x7fc588131660 con 0x7fc58813f710 2024-03-25T18:12:25.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 >> v1:172.21.15.37:6790/0 conn(0x7fc5880a98f0 legacy=0x7fc588130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:25.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 >> v1:172.21.15.37:6789/0 conn(0x7fc5881430c0 legacy=0x7fc5881e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:25.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc584ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc584ff9640 1 monclient: found mon.b 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc584ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc584ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc588131980 con 0x7fc58813f710 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc584ff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc584ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc584ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc58e3c8640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc58e3c8640 10 monclient: _renew_subs 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc58e3c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc58e3c8640 1 -- 172.21.15.37:0/3441895775 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc588135c60 con 0x7fc58813f710 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc58e3c8640 10 client.4435.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc58e3c8640 10 monclient: _renew_subs 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc58e3c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc58e3c8640 1 -- 172.21.15.37:0/3441895775 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5880b7770 con 0x7fc58813f710 2024-03-25T18:12:25.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.177+0000 7fc58e3c8640 1 librados: init done 2024-03-25T18:12:25.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.178+0000 7fc58e3c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:25.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.178+0000 7fc58e3c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:25.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.178+0000 7fc58e3c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:25.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.178+0000 7fc58e3c8640 1 -- 172.21.15.37:0/3441895775 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5881371d0 con 0x7fc58813f710 2024-03-25T18:12:25.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.452+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc57c003610 con 0x7fc58813f710 2024-03-25T18:12:25.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.452+0000 7fc584ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:25.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.452+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fc57c003920 con 0x7fc58813f710 2024-03-25T18:12:25.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.452+0000 7fc584ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:25.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.452+0000 7fc584ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:25.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.452+0000 7fc584ff9640 10 monclient: dump: 2024-03-25T18:12:25.453 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:25.453 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:25.454 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:25.454 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:25.454 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:25.454 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:25.454 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:25.454 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:25.454 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:25.454 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:25.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.452+0000 7fc5867fc640 4 set_mon_vals no callback set 2024-03-25T18:12:25.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.452+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fc57c0198f0 con 0x7fc58813f710 2024-03-25T18:12:25.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.453+0000 7fc587fff640 10 monclient: get_auth_request con 0x7fc57406e000 auth_method 0 2024-03-25T18:12:25.455 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.453+0000 7fc584ff9640 10 client.4435.objecter ms_handle_connect 0x7fc57406e000 2024-03-25T18:12:25.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.458+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fc57c0535f0 con 0x7fc58813f710 2024-03-25T18:12:25.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.458+0000 7fc584ff9640 10 client.4435.objecter ms_dispatch 0x7fc588000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:25.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.458+0000 7fc584ff9640 3 client.4435.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:25.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.458+0000 7fc584ff9640 3 client.4435.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:25.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.458+0000 7fc584ff9640 20 client.4435.objecter dump_active .. 0 homeless 2024-03-25T18:12:25.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.460+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc57c04ef30 con 0x7fc58813f710 2024-03-25T18:12:25.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.460+0000 7fc584ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:25.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.460+0000 7fc584ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:25.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.685+0000 7fc58e3c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:25.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.685+0000 7fc58e3c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:25.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.685+0000 7fc58e3c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:25.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.685+0000 7fc58e3c8640 1 -- 172.21.15.37:0/3441895775 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc588074cc0 con 0x7fc58813f710 2024-03-25T18:12:25.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.703+0000 7fc584ff9640 1 -- 172.21.15.37:0/3441895775 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fc57c051030 con 0x7fc58813f710 2024-03-25T18:12:25.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.703+0000 7fc584ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:25.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.703+0000 7fc584ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:25.704 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:12:25.704 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:25.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 10 librados: watch_flush enter 2024-03-25T18:12:25.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 10 librados: watch_flush exit 2024-03-25T18:12:25.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 20 client.4435.objecter shutdown clearing up homeless session... 2024-03-25T18:12:25.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 10 client.4435.objecter successfully canceled tick 2024-03-25T18:12:25.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 1 -- 172.21.15.37:0/3441895775 >> v1:172.21.15.117:6800/3703993921 conn(0x7fc57406e000 legacy=0x7fc5740704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:25.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 10 monclient: shutdown 2024-03-25T18:12:25.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:25.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 1 -- 172.21.15.37:0/3441895775 >> v1:172.21.15.117:6789/0 conn(0x7fc58813f710 legacy=0x7fc5881d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:25.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 1 -- 172.21.15.37:0/3441895775 shutdown_connections 2024-03-25T18:12:25.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 1 -- 172.21.15.37:0/3441895775 >> 172.21.15.37:0/3441895775 conn(0x7fc5880b7ea0 msgr2=0x7fc588074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:25.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 1 -- 172.21.15.37:0/3441895775 shutdown_connections 2024-03-25T18:12:25.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 1 -- 172.21.15.37:0/3441895775 wait complete. 2024-03-25T18:12:25.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:25.706+0000 7fc58e3c8640 1 librados: shutdown 2024-03-25T18:12:25.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:29.862 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:05:05.207 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:29.904 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:05:05.217 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:29.907 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:05:05.220 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:29.911 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:05:05.257 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:29.942 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:05:05.287 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:29.943 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:05:05.256 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:29.968 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:05:05.314 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:29.979 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:05:05.325 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:30.004 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:05:05.349 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:32.525 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:32.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.639+0000 7f1539c3e640 10 monclient: get_monmap_and_config 2024-03-25T18:12:32.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.639+0000 7f1539c3e640 10 monclient: build_initial_monmap 2024-03-25T18:12:32.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.639+0000 7f1539c3e640 10 monclient: monmap: 2024-03-25T18:12:32.641 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:32.641 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:32.641 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:12:32.640796+0000 2024-03-25T18:12:32.641 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:12:32.640796+0000 2024-03-25T18:12:32.641 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:32.641 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:32.641 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:32.641 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:12:32.641 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:12:32.642 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:32.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.640+0000 7f1539c3e640 1 Processor -- start 2024-03-25T18:12:32.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.640+0000 7f1539c3e640 1 -- start start 2024-03-25T18:12:32.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.640+0000 7f1539c3e640 10 monclient: init 2024-03-25T18:12:32.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.641+0000 7f1539c3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:32.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.641+0000 7f1539c3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:32.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.641+0000 7f1539c3e640 10 monclient(hunting): picked mon.noname-b con 0x7f15340705b0 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:32.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.641+0000 7f1539c3e640 10 monclient(hunting): picked mon.noname-c con 0x7f15341402d0 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:32.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.641+0000 7f1539c3e640 10 monclient(hunting): picked mon.noname-a con 0x7f1534149ae0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:32.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.641+0000 7f1539c3e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15340718c0 con 0x7f1534149ae0 2024-03-25T18:12:32.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.641+0000 7f1539c3e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15340714c0 con 0x7f15341402d0 2024-03-25T18:12:32.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.641+0000 7f1539c3e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1534072b50 con 0x7f15340705b0 2024-03-25T18:12:32.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.641+0000 7f1539c3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:32.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.641+0000 7f1539c3e640 10 monclient(hunting): authenticate will time out at 7107.894274s 2024-03-25T18:12:32.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.647+0000 7f1533fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f1534149ae0 0x7f153414bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52990/0 (socket says 172.21.15.37:52990) 2024-03-25T18:12:32.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.647+0000 7f1533fff640 1 -- 172.21.15.37:0/1131040269 learned_addr learned my addr 172.21.15.37:0/1131040269 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:32.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.661+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3191706429 0 0) 0x7f1534072b50 con 0x7f15340705b0 2024-03-25T18:12:32.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.661+0000 7f15327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:32.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.661+0000 7f15327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:32.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.661+0000 7f15327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:32.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.661+0000 7f15327fc640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:12:32.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.661+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f151c003520 con 0x7f15340705b0 2024-03-25T18:12:32.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.666+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1216448099 0 0) 0x7f15340718c0 con 0x7f1534149ae0 2024-03-25T18:12:32.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.666+0000 7f15327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:32.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.666+0000 7f15327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:32.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.666+0000 7f15327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:32.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.666+0000 7f15327fc640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:12:32.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.666+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1534072b50 con 0x7f1534149ae0 2024-03-25T18:12:32.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.677+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3907514749 0 0) 0x7f15340714c0 con 0x7f15341402d0 2024-03-25T18:12:32.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.677+0000 7f15327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:12:32.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.677+0000 7f15327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:32.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.677+0000 7f15327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:32.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.677+0000 7f15327fc640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:12:32.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.677+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15340718c0 con 0x7f15341402d0 2024-03-25T18:12:32.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2053762539 0 0) 0x7f15340718c0 con 0x7f15341402d0 2024-03-25T18:12:32.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 >> v1:172.21.15.37:6790/0 conn(0x7f15340705b0 legacy=0x7f153413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 >> v1:172.21.15.37:6789/0 conn(0x7f1534149ae0 legacy=0x7f153414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 1 monclient: found mon.noname-c 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f153413dd10 con 0x7f15341402d0 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 10 monclient: _finish_auth 0 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f15280015b0 con 0x7f15341402d0 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f1539c3e640 5 monclient: authenticate success, global_id 4438 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f1539c3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:32.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f15327fc640 10 monclient: dump: 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:32.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.685+0000 7f1539c3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:32.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.701+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1528001710 con 0x7f15341402d0 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.701+0000 7f15327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.701+0000 7f15327fc640 1 -- 172.21.15.37:0/1131040269 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1528001b60 con 0x7f15341402d0 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.701+0000 7f1539c3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.701+0000 7f1539c3e640 4 set_mon_vals no callback set 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.701+0000 7f15327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.701+0000 7f15327fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.701+0000 7f15327fc640 10 monclient: dump: 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:32.703 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:32.704 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:32.704 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:32.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.702+0000 7f1539c3e640 10 monclient: shutdown 2024-03-25T18:12:32.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.702+0000 7f1539c3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:32.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.702+0000 7f1539c3e640 1 -- 172.21.15.37:0/1131040269 >> v1:172.21.15.117:6789/0 conn(0x7f15341402d0 legacy=0x7f1534140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:32.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.702+0000 7f1539c3e640 1 -- 172.21.15.37:0/1131040269 shutdown_connections 2024-03-25T18:12:32.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.702+0000 7f1539c3e640 1 -- 172.21.15.37:0/1131040269 >> 172.21.15.37:0/1131040269 conn(0x7f15340b5d10 msgr2=0x7f15340b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:32.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.702+0000 7f1539c3e640 1 -- 172.21.15.37:0/1131040269 shutdown_connections 2024-03-25T18:12:32.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.702+0000 7f1539c3e640 1 -- 172.21.15.37:0/1131040269 wait complete. 2024-03-25T18:12:32.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.703+0000 7f1539c3e640 10 monclient: build_initial_monmap 2024-03-25T18:12:32.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.703+0000 7f1539c3e640 10 monclient: monmap: 2024-03-25T18:12:32.705 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:32.705 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.703+0000 7f1539c3e640 1 librados: starting msgr at 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.703+0000 7f1539c3e640 1 librados: starting objecter 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.703+0000 7f1539c3e640 1 Processor -- start 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.704+0000 7f1539c3e640 1 -- start start 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.704+0000 7f1539c3e640 1 librados: setting wanted keys 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.704+0000 7f1539c3e640 1 librados: calling monclient init 2024-03-25T18:12:32.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.704+0000 7f1539c3e640 10 monclient: init 2024-03-25T18:12:32.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.704+0000 7f1539c3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:32.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.704+0000 7f1539c3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:12:32.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.704+0000 7f1539c3e640 10 monclient(hunting): picked mon.noname-c con 0x7f15340705b0 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:32.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.704+0000 7f1539c3e640 10 monclient(hunting): picked mon.noname-b con 0x7f15341402d0 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:32.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.704+0000 7f1539c3e640 10 monclient(hunting): picked mon.noname-a con 0x7f1534149ae0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:32.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.704+0000 7f1539c3e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15341d9690 con 0x7f1534149ae0 2024-03-25T18:12:32.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.704+0000 7f1539c3e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15341d9820 con 0x7f15341402d0 2024-03-25T18:12:32.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.705+0000 7f1539c3e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1534136620 con 0x7f15340705b0 2024-03-25T18:12:32.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.705+0000 7f1539c3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:32.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.705+0000 7f1539c3e640 10 monclient(hunting): authenticate will time out at 7107.957368s 2024-03-25T18:12:32.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.713+0000 7f1532ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f15341402d0 0x7f1534137210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:54166/0 (socket says 172.21.15.37:54166) 2024-03-25T18:12:32.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.713+0000 7f1532ffd640 1 -- 172.21.15.37:0/2828770325 learned_addr learned my addr 172.21.15.37:0/2828770325 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:32.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.715+0000 7f151bfff640 10 client.?.objecter ms_handle_connect 0x7f1534149ae0 2024-03-25T18:12:32.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.715+0000 7f151bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:32.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.718+0000 7f151bfff640 10 client.?.objecter ms_handle_connect 0x7f15341402d0 2024-03-25T18:12:32.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.718+0000 7f151bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:32.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.732+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3628530830 0 0) 0x7f15341d9690 con 0x7f1534149ae0 2024-03-25T18:12:32.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.732+0000 7f151bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:32.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.732+0000 7f151bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:32.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.732+0000 7f151bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:32.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.732+0000 7f151bfff640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:12:32.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.732+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1520004eb0 con 0x7f1534149ae0 2024-03-25T18:12:32.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.733+0000 7f151bfff640 10 client.?.objecter ms_handle_connect 0x7f15340705b0 2024-03-25T18:12:32.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.733+0000 7f151bfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:32.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.740+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4058738312 0 0) 0x7f15341d9820 con 0x7f15341402d0 2024-03-25T18:12:32.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.740+0000 7f151bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:32.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.740+0000 7f151bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:32.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.740+0000 7f151bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:32.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.740+0000 7f151bfff640 10 monclient(hunting): my global_id is 4441 2024-03-25T18:12:32.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.740+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15341d9690 con 0x7f15341402d0 2024-03-25T18:12:32.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.751+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 893576681 0 0) 0x7f1534136620 con 0x7f15340705b0 2024-03-25T18:12:32.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.751+0000 7f151bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:12:32.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.751+0000 7f151bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:32.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.751+0000 7f151bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:32.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.751+0000 7f151bfff640 10 monclient(hunting): my global_id is 4436 2024-03-25T18:12:32.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.751+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15341d9820 con 0x7f15340705b0 2024-03-25T18:12:32.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.754+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3073727814 0 0) 0x7f1520004eb0 con 0x7f1534149ae0 2024-03-25T18:12:32.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.754+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1534136620 con 0x7f1534149ae0 2024-03-25T18:12:32.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.754+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f152c001bd0 con 0x7f1534149ae0 2024-03-25T18:12:32.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.754+0000 7f151bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:32.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.754+0000 7f151bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:32.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.754+0000 7f151bfff640 10 monclient(hunting): dump: 2024-03-25T18:12:32.756 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:32.756 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:32.756 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:32.756 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:32.756 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:32.756 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:32.756 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:32.756 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:32.756 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:32.756 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:32.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.754+0000 7f151bfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:32.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.759+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1089566102 0 0) 0x7f15341d9690 con 0x7f15341402d0 2024-03-25T18:12:32.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.759+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1520004eb0 con 0x7f15341402d0 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.759+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1528001aa0 con 0x7f15341402d0 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.759+0000 7f151bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.759+0000 7f151bfff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.759+0000 7f151bfff640 10 monclient(hunting): dump: 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:32.761 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2843842850 0 0) 0x7f1534136620 con 0x7f1534149ae0 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 >> v1:172.21.15.37:6790/0 conn(0x7f15340705b0 legacy=0x7f15341d8f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 >> v1:172.21.15.117:6789/0 conn(0x7f15341402d0 legacy=0x7f1534137210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f151bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f151bfff640 1 monclient: found mon.a 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f151bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f151bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15341367b0 con 0x7f1534149ae0 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f151bfff640 10 monclient: _finish_auth 0 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f151bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f151bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:32.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f1539c3e640 5 monclient: authenticate success, global_id 4440 2024-03-25T18:12:32.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f1539c3e640 10 monclient: _renew_subs 2024-03-25T18:12:32.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f1539c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:32.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f1539c3e640 1 -- 172.21.15.37:0/2828770325 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f153413dfd0 con 0x7f1534149ae0 2024-03-25T18:12:32.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f1539c3e640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:32.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f1539c3e640 10 monclient: _renew_subs 2024-03-25T18:12:32.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f1539c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:32.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f1539c3e640 1 -- 172.21.15.37:0/2828770325 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1534137b00 con 0x7f1534149ae0 2024-03-25T18:12:32.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.775+0000 7f1539c3e640 1 librados: init done 2024-03-25T18:12:32.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.777+0000 7f1539c3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:32.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.777+0000 7f1539c3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:32.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.777+0000 7f1539c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:32.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:32.777+0000 7f1539c3e640 1 -- 172.21.15.37:0/2828770325 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15341d8dd0 con 0x7f1534149ae0 2024-03-25T18:12:33.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.106+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f152c0020c0 con 0x7f1534149ae0 2024-03-25T18:12:33.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.106+0000 7f151bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:33.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.106+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f152c004d00 con 0x7f1534149ae0 2024-03-25T18:12:33.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.106+0000 7f151bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:33.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.106+0000 7f151bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:33.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.106+0000 7f151bfff640 10 monclient: dump: 2024-03-25T18:12:33.107 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:33.108 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:33.108 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:33.108 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:33.108 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:33.108 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:33.108 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:33.108 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:33.108 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:33.108 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:33.108 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.106+0000 7f1531ffb640 4 set_mon_vals no callback set 2024-03-25T18:12:33.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.107+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f152c01ad50 con 0x7f1534149ae0 2024-03-25T18:12:33.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.107+0000 7f15337fe640 10 monclient: get_auth_request con 0x7f152006df20 auth_method 0 2024-03-25T18:12:33.109 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.107+0000 7f151bfff640 10 client.4440.objecter ms_handle_connect 0x7f152006df20 2024-03-25T18:12:33.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.112+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f152c054a30 con 0x7f1534149ae0 2024-03-25T18:12:33.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.112+0000 7f151bfff640 10 client.4440.objecter ms_dispatch 0x7f1534000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:33.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.112+0000 7f151bfff640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:33.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.112+0000 7f151bfff640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:33.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.112+0000 7f151bfff640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-25T18:12:33.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.112+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f152c050370 con 0x7f1534149ae0 2024-03-25T18:12:33.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.112+0000 7f151bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:33.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.112+0000 7f151bfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:33.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.333+0000 7f1539c3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:33.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.333+0000 7f1539c3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:33.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.333+0000 7f1539c3e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:33.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.333+0000 7f1539c3e640 1 -- 172.21.15.37:0/2828770325 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15340627b0 con 0x7f1534149ae0 2024-03-25T18:12:33.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.372+0000 7f151bfff640 1 -- 172.21.15.37:0/2828770325 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f152c052470 con 0x7f1534149ae0 2024-03-25T18:12:33.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.372+0000 7f151bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:33.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.372+0000 7f151bfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:33.373 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:12:33.373 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:33.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 10 librados: watch_flush enter 2024-03-25T18:12:33.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 10 librados: watch_flush exit 2024-03-25T18:12:33.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-25T18:12:33.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 10 client.4440.objecter successfully canceled tick 2024-03-25T18:12:33.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 1 -- 172.21.15.37:0/2828770325 >> v1:172.21.15.117:6800/3703993921 conn(0x7f152006df20 legacy=0x7f15200703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:33.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 10 monclient: shutdown 2024-03-25T18:12:33.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:33.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 1 -- 172.21.15.37:0/2828770325 >> v1:172.21.15.37:6789/0 conn(0x7f1534149ae0 legacy=0x7f1534131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:33.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 1 -- 172.21.15.37:0/2828770325 shutdown_connections 2024-03-25T18:12:33.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 1 -- 172.21.15.37:0/2828770325 >> 172.21.15.37:0/2828770325 conn(0x7f15340b5d10 msgr2=0x7f153413b1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:33.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 1 -- 172.21.15.37:0/2828770325 shutdown_connections 2024-03-25T18:12:33.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 1 -- 172.21.15.37:0/2828770325 wait complete. 2024-03-25T18:12:33.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:33.374+0000 7f1539c3e640 1 librados: shutdown 2024-03-25T18:12:33.387 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:40.194 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:40.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.308+0000 7f207315d640 10 monclient: get_monmap_and_config 2024-03-25T18:12:40.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.308+0000 7f207315d640 10 monclient: build_initial_monmap 2024-03-25T18:12:40.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.308+0000 7f207315d640 10 monclient: monmap: 2024-03-25T18:12:40.309 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:40.309 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:40.310 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:12:40.309248+0000 2024-03-25T18:12:40.310 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:12:40.309248+0000 2024-03-25T18:12:40.310 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:40.310 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:40.310 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:40.310 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:12:40.310 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:12:40.310 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:40.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.309+0000 7f207315d640 1 Processor -- start 2024-03-25T18:12:40.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.309+0000 7f207315d640 1 -- start start 2024-03-25T18:12:40.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.309+0000 7f207315d640 10 monclient: init 2024-03-25T18:12:40.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.310+0000 7f207315d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:40.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.310+0000 7f207315d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:40.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.310+0000 7f207315d640 10 monclient(hunting): picked mon.noname-a con 0x7f206c0a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:40.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.310+0000 7f207315d640 10 monclient(hunting): picked mon.noname-b con 0x7f206c13f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:40.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.310+0000 7f207315d640 10 monclient(hunting): picked mon.noname-c con 0x7f206c1430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:40.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.310+0000 7f207315d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206c142520 con 0x7f206c0a98f0 2024-03-25T18:12:40.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.310+0000 7f207315d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206c0b28a0 con 0x7f206c1430c0 2024-03-25T18:12:40.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.310+0000 7f207315d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206c0b2a30 con 0x7f206c13f710 2024-03-25T18:12:40.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.310+0000 7f207315d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:40.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.310+0000 7f207315d640 10 monclient(hunting): authenticate will time out at 7115.562700s 2024-03-25T18:12:40.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.324+0000 7f206bfff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f206c13f710 0x7f206c1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:45334/0 (socket says 172.21.15.37:45334) 2024-03-25T18:12:40.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.324+0000 7f206bfff640 1 -- 172.21.15.37:0/2766235807 learned_addr learned my addr 172.21.15.37:0/2766235807 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:40.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.337+0000 7f206b7fe640 1 -- 172.21.15.37:0/2766235807 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2901559521 0 0) 0x7f206c0b2a30 con 0x7f206c13f710 2024-03-25T18:12:40.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.337+0000 7f206b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.337+0000 7f206b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.337+0000 7f206b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.337+0000 7f206b7fe640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:12:40.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.337+0000 7f206b7fe640 1 -- 172.21.15.37:0/2766235807 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f205c003520 con 0x7f206c13f710 2024-03-25T18:12:40.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.338+0000 7f206b7fe640 1 -- 172.21.15.37:0/2766235807 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1925063062 0 0) 0x7f206c0b28a0 con 0x7f206c1430c0 2024-03-25T18:12:40.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.338+0000 7f206b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.338+0000 7f206b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.338+0000 7f206b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.338+0000 7f206b7fe640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:12:40.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.338+0000 7f206b7fe640 1 -- 172.21.15.37:0/2766235807 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f206c0b2a30 con 0x7f206c1430c0 2024-03-25T18:12:40.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 1 -- 172.21.15.37:0/2766235807 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 98438336 0 0) 0x7f206c0b2a30 con 0x7f206c1430c0 2024-03-25T18:12:40.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 1 -- 172.21.15.37:0/2766235807 >> v1:172.21.15.37:6790/0 conn(0x7f206c13f710 legacy=0x7f206c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 1 -- 172.21.15.37:0/2766235807 >> v1:172.21.15.37:6789/0 conn(0x7f206c0a98f0 legacy=0x7f206c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:12:40.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 1 monclient: found mon.noname-c 2024-03-25T18:12:40.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:40.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:12:40.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 1 -- 172.21.15.37:0/2766235807 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c0b44c0 con 0x7f206c1430c0 2024-03-25T18:12:40.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:12:40.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:12:40.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:40.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 1 -- 172.21.15.37:0/2766235807 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f20640019f0 con 0x7f206c1430c0 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f206b7fe640 10 monclient: dump: 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:40.349 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:40.350 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:40.350 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f207315d640 5 monclient: authenticate success, global_id 4444 2024-03-25T18:12:40.350 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.346+0000 7f207315d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:40.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.359+0000 7f206b7fe640 1 -- 172.21.15.37:0/2766235807 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2064001b50 con 0x7f206c1430c0 2024-03-25T18:12:40.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.359+0000 7f206b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:40.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.359+0000 7f207315d640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:40.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.359+0000 7f207315d640 4 set_mon_vals no callback set 2024-03-25T18:12:40.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.360+0000 7f207315d640 10 monclient: shutdown 2024-03-25T18:12:40.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.360+0000 7f207315d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:40.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.360+0000 7f207315d640 1 -- 172.21.15.37:0/2766235807 >> v1:172.21.15.117:6789/0 conn(0x7f206c1430c0 legacy=0x7f206c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.360+0000 7f207315d640 1 -- 172.21.15.37:0/2766235807 shutdown_connections 2024-03-25T18:12:40.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.360+0000 7f207315d640 1 -- 172.21.15.37:0/2766235807 >> 172.21.15.37:0/2766235807 conn(0x7f206c0b7ea0 msgr2=0x7f206c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:40.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.360+0000 7f207315d640 1 -- 172.21.15.37:0/2766235807 shutdown_connections 2024-03-25T18:12:40.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.360+0000 7f207315d640 1 -- 172.21.15.37:0/2766235807 wait complete. 2024-03-25T18:12:40.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.361+0000 7f207315d640 10 monclient: build_initial_monmap 2024-03-25T18:12:40.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.361+0000 7f207315d640 10 monclient: monmap: 2024-03-25T18:12:40.362 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:40.362 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:40.362 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:12:40.362 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:12:40.363 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:40.363 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:12:40.363 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:40.363 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:12:40.363 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:12:40.363 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:40.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.361+0000 7f207315d640 1 librados: starting msgr at 2024-03-25T18:12:40.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.361+0000 7f207315d640 1 librados: starting objecter 2024-03-25T18:12:40.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.362+0000 7f207315d640 1 Processor -- start 2024-03-25T18:12:40.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.362+0000 7f207315d640 1 -- start start 2024-03-25T18:12:40.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.362+0000 7f207315d640 1 librados: setting wanted keys 2024-03-25T18:12:40.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.362+0000 7f207315d640 1 librados: calling monclient init 2024-03-25T18:12:40.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.362+0000 7f207315d640 10 monclient: init 2024-03-25T18:12:40.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.363+0000 7f207315d640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:40.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.363+0000 7f207315d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:40.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.363+0000 7f207315d640 10 monclient(hunting): picked mon.noname-a con 0x7f206c0a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:40.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.363+0000 7f207315d640 10 monclient(hunting): picked mon.noname-b con 0x7f206c13f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:40.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.364+0000 7f207315d640 10 monclient(hunting): picked mon.noname-c con 0x7f206c1430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:40.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.364+0000 7f207315d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206c1dee20 con 0x7f206c0a98f0 2024-03-25T18:12:40.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.364+0000 7f207315d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206c1dea40 con 0x7f206c13f710 2024-03-25T18:12:40.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.364+0000 7f207315d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f206c1debd0 con 0x7f206c1430c0 2024-03-25T18:12:40.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.364+0000 7f207315d640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:40.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.364+0000 7f207315d640 10 monclient(hunting): authenticate will time out at 7115.616855s 2024-03-25T18:12:40.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.378+0000 7f206bfff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f206c13f710 0x7f206c1df340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:50296/0 (socket says 172.21.15.37:50296) 2024-03-25T18:12:40.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.378+0000 7f206bfff640 1 -- 172.21.15.37:0/270556303 learned_addr learned my addr 172.21.15.37:0/270556303 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:40.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.380+0000 7f20697fa640 10 client.?.objecter ms_handle_connect 0x7f206c13f710 2024-03-25T18:12:40.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.380+0000 7f20697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.382+0000 7f20697fa640 10 client.?.objecter ms_handle_connect 0x7f206c1430c0 2024-03-25T18:12:40.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.382+0000 7f20697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.387+0000 7f20697fa640 10 client.?.objecter ms_handle_connect 0x7f206c0a98f0 2024-03-25T18:12:40.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.387+0000 7f20697fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:40.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.388+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1819243141 0 0) 0x7f206c1dea40 con 0x7f206c13f710 2024-03-25T18:12:40.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.388+0000 7f20697fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.388+0000 7f20697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.388+0000 7f20697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.388+0000 7f20697fa640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:12:40.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.389+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2050004eb0 con 0x7f206c13f710 2024-03-25T18:12:40.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.392+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3968155210 0 0) 0x7f206c1debd0 con 0x7f206c1430c0 2024-03-25T18:12:40.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.392+0000 7f20697fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.392+0000 7f20697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.392+0000 7f20697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.392+0000 7f20697fa640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:12:40.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.392+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f206c1dea40 con 0x7f206c1430c0 2024-03-25T18:12:40.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.394+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4085584562 0 0) 0x7f206c1dee20 con 0x7f206c0a98f0 2024-03-25T18:12:40.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.394+0000 7f20697fa640 10 monclient(hunting): _negotiate 2024-03-25T18:12:40.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.394+0000 7f20697fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:40.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.394+0000 7f20697fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:40.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.394+0000 7f20697fa640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:12:40.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.394+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f206c1debd0 con 0x7f206c0a98f0 2024-03-25T18:12:40.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.406+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2870453584 0 0) 0x7f2050004eb0 con 0x7f206c13f710 2024-03-25T18:12:40.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.406+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f206c1dee20 con 0x7f206c13f710 2024-03-25T18:12:40.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.406+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f2058001990 con 0x7f206c13f710 2024-03-25T18:12:40.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.406+0000 7f20697fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.406+0000 7f20697fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:40.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.406+0000 7f20697fa640 10 monclient(hunting): dump: 2024-03-25T18:12:40.407 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:40.407 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:40.407 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:40.408 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:40.408 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.408 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:40.408 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:40.408 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:40.408 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:40.408 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:40.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.406+0000 7f20697fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:40.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4275590964 0 0) 0x7f206c1dee20 con 0x7f206c13f710 2024-03-25T18:12:40.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 >> v1:172.21.15.37:6790/0 conn(0x7f206c1430c0 legacy=0x7f206c137fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 >> v1:172.21.15.37:6789/0 conn(0x7f206c0a98f0 legacy=0x7f206c1db540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:40.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f20697fa640 10 monclient: _finish_hunting 0 2024-03-25T18:12:40.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f20697fa640 1 monclient: found mon.b 2024-03-25T18:12:40.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f20697fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:40.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f20697fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:40.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f206c06dea0 con 0x7f206c13f710 2024-03-25T18:12:40.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f20697fa640 10 monclient: _finish_auth 0 2024-03-25T18:12:40.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f20697fa640 10 monclient: _check_auth_tickets 2024-03-25T18:12:40.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f20697fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:40.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f207315d640 5 monclient: authenticate success, global_id 4447 2024-03-25T18:12:40.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f207315d640 10 monclient: _renew_subs 2024-03-25T18:12:40.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f207315d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:40.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.413+0000 7f207315d640 1 -- 172.21.15.37:0/270556303 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f206c06e0f0 con 0x7f206c13f710 2024-03-25T18:12:40.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.414+0000 7f207315d640 10 client.4447.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:40.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.414+0000 7f207315d640 10 monclient: _renew_subs 2024-03-25T18:12:40.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.414+0000 7f207315d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:40.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.414+0000 7f207315d640 1 -- 172.21.15.37:0/270556303 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f206c071970 con 0x7f206c13f710 2024-03-25T18:12:40.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.414+0000 7f207315d640 1 librados: init done 2024-03-25T18:12:40.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.415+0000 7f207315d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.415+0000 7f207315d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.415+0000 7f207315d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:40.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.415+0000 7f207315d640 1 -- 172.21.15.37:0/270556303 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f206c0627b0 con 0x7f206c13f710 2024-03-25T18:12:40.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.720+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20580047e0 con 0x7f206c13f710 2024-03-25T18:12:40.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.720+0000 7f20697fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:40.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.720+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f2058004bb0 con 0x7f206c13f710 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.720+0000 7f20697fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.720+0000 7f20697fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.720+0000 7f20697fa640 10 monclient: dump: 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:40.722 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:40.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.720+0000 7f206affd640 4 set_mon_vals no callback set 2024-03-25T18:12:40.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.721+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f2058004dd0 con 0x7f206c13f710 2024-03-25T18:12:40.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.722+0000 7f2070ef9640 10 monclient: get_auth_request con 0x7f205006ddf0 auth_method 0 2024-03-25T18:12:40.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.722+0000 7f20697fa640 10 client.4447.objecter ms_handle_connect 0x7f205006ddf0 2024-03-25T18:12:40.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.727+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f20580548f0 con 0x7f206c13f710 2024-03-25T18:12:40.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.727+0000 7f20697fa640 10 client.4447.objecter ms_dispatch 0x7f206c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:40.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.727+0000 7f20697fa640 3 client.4447.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:40.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.727+0000 7f20697fa640 3 client.4447.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:40.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.727+0000 7f20697fa640 20 client.4447.objecter dump_active .. 0 homeless 2024-03-25T18:12:40.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.727+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2058050400 con 0x7f206c13f710 2024-03-25T18:12:40.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.727+0000 7f20697fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:40.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.727+0000 7f20697fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:40.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.973+0000 7f207315d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:40.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.974+0000 7f207315d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:40.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.974+0000 7f207315d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:40.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:40.974+0000 7f207315d640 1 -- 172.21.15.37:0/270556303 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f206c1defb0 con 0x7f206c13f710 2024-03-25T18:12:41.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.045+0000 7f20697fa640 1 -- 172.21.15.37:0/270556303 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f2058052330 con 0x7f206c13f710 2024-03-25T18:12:41.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.045+0000 7f20697fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:41.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.045+0000 7f20697fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:41.047 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:12:41.047 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:41.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 10 librados: watch_flush enter 2024-03-25T18:12:41.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 10 librados: watch_flush exit 2024-03-25T18:12:41.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 20 client.4447.objecter shutdown clearing up homeless session... 2024-03-25T18:12:41.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 10 client.4447.objecter successfully canceled tick 2024-03-25T18:12:41.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 1 -- 172.21.15.37:0/270556303 >> v1:172.21.15.117:6800/3703993921 conn(0x7f205006ddf0 legacy=0x7f20500702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:41.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 10 monclient: shutdown 2024-03-25T18:12:41.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:41.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 1 -- 172.21.15.37:0/270556303 >> v1:172.21.15.117:6789/0 conn(0x7f206c13f710 legacy=0x7f206c1df340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:41.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 1 -- 172.21.15.37:0/270556303 shutdown_connections 2024-03-25T18:12:41.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 1 -- 172.21.15.37:0/270556303 >> 172.21.15.37:0/270556303 conn(0x7f206c0b7ea0 msgr2=0x7f206c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:41.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 1 -- 172.21.15.37:0/270556303 shutdown_connections 2024-03-25T18:12:41.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 1 -- 172.21.15.37:0/270556303 wait complete. 2024-03-25T18:12:41.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:41.048+0000 7f207315d640 1 librados: shutdown 2024-03-25T18:12:41.062 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:47.868 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:47.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.980+0000 7f13c2358640 10 monclient: get_monmap_and_config 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.980+0000 7f13c2358640 10 monclient: build_initial_monmap 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.980+0000 7f13c2358640 10 monclient: monmap: 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:12:47.981711+0000 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:12:47.981711+0000 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:12:47.982 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:47.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.981+0000 7f13c2358640 1 Processor -- start 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.981+0000 7f13c2358640 1 -- start start 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.981+0000 7f13c2358640 10 monclient: init 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.982+0000 7f13c2358640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.982+0000 7f13c2358640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.982+0000 7f13c2358640 10 monclient(hunting): picked mon.noname-a con 0x7f13bc0a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.982+0000 7f13c2358640 10 monclient(hunting): picked mon.noname-c con 0x7f13bc13f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.982+0000 7f13c2358640 10 monclient(hunting): picked mon.noname-b con 0x7f13bc1430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.982+0000 7f13c2358640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc142520 con 0x7f13bc0a98f0 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.982+0000 7f13c2358640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0b28a0 con 0x7f13bc13f710 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.982+0000 7f13c2358640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc0b2a30 con 0x7f13bc1430c0 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.982+0000 7f13c2358640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:47.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.982+0000 7f13c2358640 10 monclient(hunting): authenticate will time out at 7123.235095s 2024-03-25T18:12:47.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.992+0000 7f13bb7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f13bc13f710 0x7f13bc1429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:40032/0 (socket says 172.21.15.37:40032) 2024-03-25T18:12:47.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:47.992+0000 7f13bb7fe640 1 -- 172.21.15.37:0/1875083 learned_addr learned my addr 172.21.15.37:0/1875083 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:48.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.007+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 435496129 0 0) 0x7f13bc0b28a0 con 0x7f13bc13f710 2024-03-25T18:12:48.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.007+0000 7f13baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:48.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.007+0000 7f13baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:48.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.007+0000 7f13baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:48.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.007+0000 7f13baffd640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:12:48.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.007+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a4003520 con 0x7f13bc13f710 2024-03-25T18:12:48.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.011+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1049317241 0 0) 0x7f13bc0b2a30 con 0x7f13bc1430c0 2024-03-25T18:12:48.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.011+0000 7f13baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:48.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.011+0000 7f13baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:48.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.011+0000 7f13baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:48.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.011+0000 7f13baffd640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:12:48.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.011+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc0b28a0 con 0x7f13bc1430c0 2024-03-25T18:12:48.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.012+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2416088802 0 0) 0x7f13bc142520 con 0x7f13bc0a98f0 2024-03-25T18:12:48.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.012+0000 7f13baffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:48.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.012+0000 7f13baffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:48.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.012+0000 7f13baffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:48.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.012+0000 7f13baffd640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:12:48.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.012+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc0b2a30 con 0x7f13bc0a98f0 2024-03-25T18:12:48.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2717681935 0 0) 0x7f13bc0b28a0 con 0x7f13bc1430c0 2024-03-25T18:12:48.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 >> v1:172.21.15.117:6789/0 conn(0x7f13bc13f710 legacy=0x7f13bc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:48.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 >> v1:172.21.15.37:6789/0 conn(0x7f13bc0a98f0 legacy=0x7f13bc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:48.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:48.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 1 monclient: found mon.noname-b 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc0b44c0 con 0x7f13bc1430c0 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 10 monclient: _finish_auth 0 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f13b40019f0 con 0x7f13bc1430c0 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13c2358640 5 monclient: authenticate success, global_id 4445 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13c2358640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13baffd640 10 monclient: dump: 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.020 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.021 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:48.021 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:48.021 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:48.021 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:48.021 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:48.021 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:48.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.018+0000 7f13c2358640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:48.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.042+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13b4001b50 con 0x7f13bc1430c0 2024-03-25T18:12:48.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.042+0000 7f13baffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.042+0000 7f13baffd640 1 -- 172.21.15.37:0/1875083 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f13b4001f60 con 0x7f13bc1430c0 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.042+0000 7f13baffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.042+0000 7f13baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.042+0000 7f13baffd640 10 monclient: dump: 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:48.044 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:48.045 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:48.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.042+0000 7f13c2358640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:48.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.042+0000 7f13c2358640 4 set_mon_vals no callback set 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.043+0000 7f13c2358640 10 monclient: shutdown 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.043+0000 7f13c2358640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.043+0000 7f13c2358640 1 -- 172.21.15.37:0/1875083 >> v1:172.21.15.37:6790/0 conn(0x7f13bc1430c0 legacy=0x7f13bc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.043+0000 7f13c2358640 1 -- 172.21.15.37:0/1875083 shutdown_connections 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.043+0000 7f13c2358640 1 -- 172.21.15.37:0/1875083 >> 172.21.15.37:0/1875083 conn(0x7f13bc0b7ea0 msgr2=0x7f13bc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.043+0000 7f13c2358640 1 -- 172.21.15.37:0/1875083 shutdown_connections 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.043+0000 7f13c2358640 1 -- 172.21.15.37:0/1875083 wait complete. 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.044+0000 7f13c2358640 10 monclient: build_initial_monmap 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.044+0000 7f13c2358640 10 monclient: monmap: 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:48.047 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.044+0000 7f13c2358640 1 librados: starting msgr at 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.044+0000 7f13c2358640 1 librados: starting objecter 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.045+0000 7f13c2358640 1 Processor -- start 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.045+0000 7f13c2358640 1 -- start start 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.045+0000 7f13c2358640 1 librados: setting wanted keys 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.045+0000 7f13c2358640 1 librados: calling monclient init 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.045+0000 7f13c2358640 10 monclient: init 2024-03-25T18:12:48.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.045+0000 7f13c2358640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:48.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.045+0000 7f13c2358640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:48.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.045+0000 7f13c2358640 10 monclient(hunting): picked mon.noname-b con 0x7f13bc0a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:48.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.046+0000 7f13c2358640 10 monclient(hunting): picked mon.noname-c con 0x7f13bc13f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:48.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.046+0000 7f13c2358640 10 monclient(hunting): picked mon.noname-a con 0x7f13bc1430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:48.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.046+0000 7f13c2358640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc1397d0 con 0x7f13bc1430c0 2024-03-25T18:12:48.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.046+0000 7f13c2358640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc13af80 con 0x7f13bc0a98f0 2024-03-25T18:12:48.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.046+0000 7f13c2358640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13bc1dba00 con 0x7f13bc13f710 2024-03-25T18:12:48.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.046+0000 7f13c2358640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:48.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.046+0000 7f13c2358640 10 monclient(hunting): authenticate will time out at 7123.298474s 2024-03-25T18:12:48.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.059+0000 7f13bb7fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f13bc13f710 0x7f13bc1dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:60636/0 (socket says 172.21.15.37:60636) 2024-03-25T18:12:48.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.059+0000 7f13bb7fe640 1 -- 172.21.15.37:0/345155384 learned_addr learned my addr 172.21.15.37:0/345155384 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:48.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.064+0000 7f13b8ff9640 10 client.?.objecter ms_handle_connect 0x7f13bc13f710 2024-03-25T18:12:48.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.064+0000 7f13b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:48.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.064+0000 7f13b8ff9640 10 client.?.objecter ms_handle_connect 0x7f13bc0a98f0 2024-03-25T18:12:48.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.064+0000 7f13b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:48.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.069+0000 7f13b8ff9640 10 client.?.objecter ms_handle_connect 0x7f13bc1430c0 2024-03-25T18:12:48.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.069+0000 7f13b8ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:48.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.076+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2060938737 0 0) 0x7f13bc1dba00 con 0x7f13bc13f710 2024-03-25T18:12:48.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.076+0000 7f13b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:48.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.076+0000 7f13b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:48.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.076+0000 7f13b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:48.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.076+0000 7f13b8ff9640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:12:48.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.076+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13a8004eb0 con 0x7f13bc13f710 2024-03-25T18:12:48.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3237378018 0 0) 0x7f13bc1397d0 con 0x7f13bc1430c0 2024-03-25T18:12:48.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:48.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:48.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:48.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:12:48.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc1dba00 con 0x7f13bc1430c0 2024-03-25T18:12:48.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 861904179 0 0) 0x7f13bc13af80 con 0x7f13bc0a98f0 2024-03-25T18:12:48.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:12:48.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:48.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:48.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:12:48.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.084+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13bc1397d0 con 0x7f13bc0a98f0 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.101+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2129324703 0 0) 0x7f13bc1dba00 con 0x7f13bc1430c0 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.101+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13bc13af80 con 0x7f13bc1430c0 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.101+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f13b4001e60 con 0x7f13bc1430c0 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.101+0000 7f13b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.101+0000 7f13b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.101+0000 7f13b8ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:48.103 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:48.104 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:48.104 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:48.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.101+0000 7f13b8ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:48.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.103+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1302527059 0 0) 0x7f13bc1397d0 con 0x7f13bc0a98f0 2024-03-25T18:12:48.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.103+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13bc1dba00 con 0x7f13bc0a98f0 2024-03-25T18:12:48.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.103+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f13ac0018f0 con 0x7f13bc0a98f0 2024-03-25T18:12:48.104 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.103+0000 7f13b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.103+0000 7f13b8ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.103+0000 7f13b8ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:48.105 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:48.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.106+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1778109527 0 0) 0x7f13a8004eb0 con 0x7f13bc13f710 2024-03-25T18:12:48.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.106+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13bc1397d0 con 0x7f13bc13f710 2024-03-25T18:12:48.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.106+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f13b0001950 con 0x7f13bc13f710 2024-03-25T18:12:48.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.106+0000 7f13b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:48.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.106+0000 7f13b8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:12:48.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.106+0000 7f13b8ff9640 10 monclient(hunting): dump: 2024-03-25T18:12:48.108 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:48.108 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:48.108 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.108 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.108 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:48.108 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:48.108 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:48.108 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:48.108 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:48.108 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:48.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1102388924 0 0) 0x7f13bc13af80 con 0x7f13bc1430c0 2024-03-25T18:12:48.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 >> v1:172.21.15.37:6790/0 conn(0x7f13bc13f710 legacy=0x7f13bc1dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:48.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 >> v1:172.21.15.117:6789/0 conn(0x7f13bc0a98f0 legacy=0x7f13bc0b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13b8ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13b8ff9640 1 monclient: found mon.a 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13b8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13bc13b260 con 0x7f13bc1430c0 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13b8ff9640 10 monclient: _finish_auth 0 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13b8ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13c2358640 5 monclient: authenticate success, global_id 4452 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13c2358640 10 monclient: _renew_subs 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13c2358640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13c2358640 1 -- 172.21.15.37:0/345155384 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc0b74f0 con 0x7f13bc1430c0 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13c2358640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:48.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13c2358640 10 monclient: _renew_subs 2024-03-25T18:12:48.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13c2358640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:48.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13c2358640 1 -- 172.21.15.37:0/345155384 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc12f780 con 0x7f13bc1430c0 2024-03-25T18:12:48.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.112+0000 7f13c2358640 1 librados: init done 2024-03-25T18:12:48.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.113+0000 7f13c2358640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:48.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.113+0000 7f13c2358640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:48.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.113+0000 7f13c2358640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:48.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.113+0000 7f13c2358640 1 -- 172.21.15.37:0/345155384 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc0627b0 con 0x7f13bc1430c0 2024-03-25T18:12:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.426+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13b4002020 con 0x7f13bc1430c0 2024-03-25T18:12:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.426+0000 7f13b8ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.426+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f13b4003dc0 con 0x7f13bc1430c0 2024-03-25T18:12:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.426+0000 7f13b8ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.426+0000 7f13b8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:12:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.426+0000 7f13b8ff9640 10 monclient: dump: 2024-03-25T18:12:48.428 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:48.428 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:48.428 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.428 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:48.428 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:48.429 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:48.429 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:48.429 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:48.429 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:48.429 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:48.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.426+0000 7f13ba7fc640 4 set_mon_vals no callback set 2024-03-25T18:12:48.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.427+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f13b4019d90 con 0x7f13bc1430c0 2024-03-25T18:12:48.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.428+0000 7f13bbfff640 10 monclient: get_auth_request con 0x7f13a806e120 auth_method 0 2024-03-25T18:12:48.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.428+0000 7f13b8ff9640 10 client.4452.objecter ms_handle_connect 0x7f13a806e120 2024-03-25T18:12:48.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.433+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f13b40538e0 con 0x7f13bc1430c0 2024-03-25T18:12:48.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.433+0000 7f13b8ff9640 10 client.4452.objecter ms_dispatch 0x7f13bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:48.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.433+0000 7f13b8ff9640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:48.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.433+0000 7f13b8ff9640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:48.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.433+0000 7f13b8ff9640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-25T18:12:48.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.433+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13b404f3f0 con 0x7f13bc1430c0 2024-03-25T18:12:48.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.433+0000 7f13b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:48.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.433+0000 7f13b8ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:48.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.667+0000 7f13c2358640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:48.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.667+0000 7f13c2358640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:48.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.667+0000 7f13c2358640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:12:48.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.667+0000 7f13c2358640 1 -- 172.21.15.37:0/345155384 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f13bc139980 con 0x7f13bc1430c0 2024-03-25T18:12:48.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.704+0000 7f13b8ff9640 1 -- 172.21.15.37:0/345155384 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f13bc139980 con 0x7f13bc1430c0 2024-03-25T18:12:48.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.704+0000 7f13b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:48.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.704+0000 7f13b8ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:48.706 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:12:48.706 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:48.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.706+0000 7f13c2358640 10 librados: watch_flush enter 2024-03-25T18:12:48.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.706+0000 7f13c2358640 10 librados: watch_flush exit 2024-03-25T18:12:48.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.706+0000 7f13c2358640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-25T18:12:48.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.706+0000 7f13c2358640 10 client.4452.objecter successfully canceled tick 2024-03-25T18:12:48.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.706+0000 7f13c2358640 1 -- 172.21.15.37:0/345155384 >> v1:172.21.15.117:6800/3703993921 conn(0x7f13a806e120 legacy=0x7f13a80705e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:48.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.706+0000 7f13c2358640 10 monclient: shutdown 2024-03-25T18:12:48.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.706+0000 7f13c2358640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:48.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.706+0000 7f13c2358640 1 -- 172.21.15.37:0/345155384 >> v1:172.21.15.37:6789/0 conn(0x7f13bc1430c0 legacy=0x7f13bc1db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:48.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.706+0000 7f13c2358640 1 -- 172.21.15.37:0/345155384 shutdown_connections 2024-03-25T18:12:48.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.707+0000 7f13c2358640 1 -- 172.21.15.37:0/345155384 >> 172.21.15.37:0/345155384 conn(0x7f13bc0b7ea0 msgr2=0x7f13bc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:48.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.707+0000 7f13c2358640 1 -- 172.21.15.37:0/345155384 shutdown_connections 2024-03-25T18:12:48.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.707+0000 7f13c2358640 1 -- 172.21.15.37:0/345155384 wait complete. 2024-03-25T18:12:48.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:48.707+0000 7f13c2358640 1 librados: shutdown 2024-03-25T18:12:48.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:12:50.001 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:05:25.346 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.050 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:05:25.363 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.052 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:05:25.397 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.052 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:05:25.365 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.064 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:05:25.377 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.098 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:05:25.444 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.125 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:05:25.470 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.133 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:05:25.478 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:50.159 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:05:25.504 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:12:55.526 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:12:55.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.638+0000 7f43f07d0640 10 monclient: get_monmap_and_config 2024-03-25T18:12:55.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.638+0000 7f43f07d0640 10 monclient: build_initial_monmap 2024-03-25T18:12:55.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.638+0000 7f43f07d0640 10 monclient: monmap: 2024-03-25T18:12:55.640 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:55.640 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:55.640 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:12:55.639600+0000 2024-03-25T18:12:55.640 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:12:55.639600+0000 2024-03-25T18:12:55.640 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:55.640 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:55.640 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:55.640 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:12:55.640 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:12:55.640 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:55.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.639+0000 7f43f07d0640 1 Processor -- start 2024-03-25T18:12:55.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.639+0000 7f43f07d0640 1 -- start start 2024-03-25T18:12:55.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.639+0000 7f43f07d0640 10 monclient: init 2024-03-25T18:12:55.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.640+0000 7f43f07d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:55.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.640+0000 7f43f07d0640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:12:55.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.640+0000 7f43f07d0640 10 monclient(hunting): picked mon.noname-c con 0x7f43e80a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:55.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.640+0000 7f43f07d0640 10 monclient(hunting): picked mon.noname-b con 0x7f43e813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:55.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.640+0000 7f43f07d0640 10 monclient(hunting): picked mon.noname-a con 0x7f43e81430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:55.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.640+0000 7f43f07d0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e8142520 con 0x7f43e81430c0 2024-03-25T18:12:55.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.640+0000 7f43f07d0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e80b28a0 con 0x7f43e80a98f0 2024-03-25T18:12:55.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.640+0000 7f43f07d0640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e80b2a30 con 0x7f43e813f710 2024-03-25T18:12:55.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.640+0000 7f43f07d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:55.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.640+0000 7f43f07d0640 10 monclient(hunting): authenticate will time out at 7130.892602s 2024-03-25T18:12:55.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.646+0000 7f43ee56c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f43e80a98f0 0x7f43e80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:44548/0 (socket says 172.21.15.37:44548) 2024-03-25T18:12:55.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.646+0000 7f43ee56c640 1 -- 172.21.15.37:0/83817578 learned_addr learned my addr 172.21.15.37:0/83817578 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:55.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.663+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1051219651 0 0) 0x7f43e8142520 con 0x7f43e81430c0 2024-03-25T18:12:55.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.663+0000 7f43ed56a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:55.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.663+0000 7f43ed56a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:55.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.663+0000 7f43ed56a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:55.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.663+0000 7f43ed56a640 10 monclient(hunting): my global_id is 4455 2024-03-25T18:12:55.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.663+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43d4003520 con 0x7f43e81430c0 2024-03-25T18:12:55.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.665+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321953318 0 0) 0x7f43e80b28a0 con 0x7f43e80a98f0 2024-03-25T18:12:55.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.665+0000 7f43ed56a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:55.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.665+0000 7f43ed56a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:55.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.665+0000 7f43ed56a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:55.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.665+0000 7f43ed56a640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:12:55.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.665+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e8142520 con 0x7f43e80a98f0 2024-03-25T18:12:55.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.677+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2224353871 0 0) 0x7f43e80b2a30 con 0x7f43e813f710 2024-03-25T18:12:55.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.677+0000 7f43ed56a640 10 monclient(hunting): _negotiate 2024-03-25T18:12:55.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.677+0000 7f43ed56a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:55.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.677+0000 7f43ed56a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:55.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.677+0000 7f43ed56a640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:12:55.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.677+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e80b28a0 con 0x7f43e813f710 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2946064267 0 0) 0x7f43e8142520 con 0x7f43e80a98f0 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 >> v1:172.21.15.37:6790/0 conn(0x7f43e813f710 legacy=0x7f43e81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 >> v1:172.21.15.37:6789/0 conn(0x7f43e81430c0 legacy=0x7f43e814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 10 monclient: _finish_hunting 0 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 1 monclient: found mon.noname-c 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e80b44c0 con 0x7f43e80a98f0 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 10 monclient: _finish_auth 0 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 10 monclient: _check_auth_tickets 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:55.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f43e4001670 con 0x7f43e80a98f0 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43ed56a640 10 monclient: dump: 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43f07d0640 5 monclient: authenticate success, global_id 4456 2024-03-25T18:12:55.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.678+0000 7f43f07d0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:12:55.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.694+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43e40017d0 con 0x7f43e80a98f0 2024-03-25T18:12:55.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.694+0000 7f43ed56a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:55.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.694+0000 7f43ed56a640 1 -- 172.21.15.37:0/83817578 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f43e4001be0 con 0x7f43e80a98f0 2024-03-25T18:12:55.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.694+0000 7f43f07d0640 10 monclient: get_monmap_and_config success 2024-03-25T18:12:55.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.694+0000 7f43f07d0640 4 set_mon_vals no callback set 2024-03-25T18:12:55.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.694+0000 7f43ed56a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.694+0000 7f43ed56a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.694+0000 7f43ed56a640 10 monclient: dump: 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:55.696 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:55.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.695+0000 7f43f07d0640 10 monclient: shutdown 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.695+0000 7f43f07d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.695+0000 7f43f07d0640 1 -- 172.21.15.37:0/83817578 >> v1:172.21.15.117:6789/0 conn(0x7f43e80a98f0 legacy=0x7f43e80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.695+0000 7f43f07d0640 1 -- 172.21.15.37:0/83817578 shutdown_connections 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.695+0000 7f43f07d0640 1 -- 172.21.15.37:0/83817578 >> 172.21.15.37:0/83817578 conn(0x7f43e80b7ea0 msgr2=0x7f43e80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.695+0000 7f43f07d0640 1 -- 172.21.15.37:0/83817578 shutdown_connections 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.695+0000 7f43f07d0640 1 -- 172.21.15.37:0/83817578 wait complete. 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.696+0000 7f43f07d0640 10 monclient: build_initial_monmap 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.696+0000 7f43f07d0640 10 monclient: monmap: 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:12:55.698 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.696+0000 7f43f07d0640 1 librados: starting msgr at 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.696+0000 7f43f07d0640 1 librados: starting objecter 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.696+0000 7f43f07d0640 1 Processor -- start 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 1 -- start start 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 1 librados: setting wanted keys 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 1 librados: calling monclient init 2024-03-25T18:12:55.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 10 monclient: init 2024-03-25T18:12:55.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 10 monclient: _reopen_session rank -1 2024-03-25T18:12:55.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:12:55.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 10 monclient(hunting): picked mon.noname-a con 0x7f43e80a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:12:55.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 10 monclient(hunting): picked mon.noname-c con 0x7f43e813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:12:55.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 10 monclient(hunting): picked mon.noname-b con 0x7f43e81430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:12:55.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e813b180 con 0x7f43e80a98f0 2024-03-25T18:12:55.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e81db890 con 0x7f43e81430c0 2024-03-25T18:12:55.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43e80b7b90 con 0x7f43e813f710 2024-03-25T18:12:55.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 10 monclient(hunting): _renew_subs 2024-03-25T18:12:55.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.697+0000 7f43f07d0640 10 monclient(hunting): authenticate will time out at 7130.950260s 2024-03-25T18:12:55.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.703+0000 7f43eed6d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f43e81430c0 0x7f43e81db180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:44596/0 (socket says 172.21.15.37:44596) 2024-03-25T18:12:55.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.703+0000 7f43eed6d640 1 -- 172.21.15.37:0/589896605 learned_addr learned my addr 172.21.15.37:0/589896605 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:12:55.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.705+0000 7f43deffd640 10 client.?.objecter ms_handle_connect 0x7f43e81430c0 2024-03-25T18:12:55.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.705+0000 7f43deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:55.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.708+0000 7f43deffd640 10 client.?.objecter ms_handle_connect 0x7f43e813f710 2024-03-25T18:12:55.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.708+0000 7f43deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:55.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.713+0000 7f43deffd640 10 client.?.objecter ms_handle_connect 0x7f43e80a98f0 2024-03-25T18:12:55.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.713+0000 7f43deffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:12:55.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.717+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585714385 0 0) 0x7f43e81db890 con 0x7f43e81430c0 2024-03-25T18:12:55.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.717+0000 7f43deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:55.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.717+0000 7f43deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:55.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.717+0000 7f43deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:55.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.717+0000 7f43deffd640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:12:55.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.717+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43cc004eb0 con 0x7f43e81430c0 2024-03-25T18:12:55.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.720+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3454705892 0 0) 0x7f43e813b180 con 0x7f43e80a98f0 2024-03-25T18:12:55.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.720+0000 7f43deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:55.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.720+0000 7f43deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:55.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.720+0000 7f43deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:55.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.720+0000 7f43deffd640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:12:55.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.720+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e81db890 con 0x7f43e80a98f0 2024-03-25T18:12:55.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.725+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448477351 0 0) 0x7f43e80b7b90 con 0x7f43e813f710 2024-03-25T18:12:55.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.725+0000 7f43deffd640 10 monclient(hunting): _negotiate 2024-03-25T18:12:55.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.725+0000 7f43deffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:12:55.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.725+0000 7f43deffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:12:55.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.725+0000 7f43deffd640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:12:55.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.725+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f43e813b180 con 0x7f43e813f710 2024-03-25T18:12:55.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.727+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 580831240 0 0) 0x7f43cc004eb0 con 0x7f43e81430c0 2024-03-25T18:12:55.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.727+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f43e80b7b90 con 0x7f43e81430c0 2024-03-25T18:12:55.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.727+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f43e0001c30 con 0x7f43e81430c0 2024-03-25T18:12:55.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.727+0000 7f43deffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.727+0000 7f43deffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.727+0000 7f43deffd640 10 monclient(hunting): dump: 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:55.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.727+0000 7f43deffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:12:55.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.733+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2841194285 0 0) 0x7f43e80b7b90 con 0x7f43e81430c0 2024-03-25T18:12:55.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 >> v1:172.21.15.37:6790/0 conn(0x7f43e813f710 legacy=0x7f43e81daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:55.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 >> v1:172.21.15.37:6789/0 conn(0x7f43e80a98f0 legacy=0x7f43e80b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:55.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43deffd640 10 monclient: _finish_hunting 0 2024-03-25T18:12:55.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43deffd640 1 monclient: found mon.b 2024-03-25T18:12:55.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43deffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:12:55.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43deffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:55.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43e813b3f0 con 0x7f43e81430c0 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43deffd640 10 monclient: _finish_auth 0 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43deffd640 10 monclient: _check_auth_tickets 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43deffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43f07d0640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43f07d0640 10 monclient: _renew_subs 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43f07d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43f07d0640 1 -- 172.21.15.37:0/589896605 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43e813a1b0 con 0x7f43e81430c0 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43f07d0640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43f07d0640 10 monclient: _renew_subs 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43f07d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43f07d0640 1 -- 172.21.15.37:0/589896605 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f43e813ba40 con 0x7f43e81430c0 2024-03-25T18:12:55.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.734+0000 7f43f07d0640 1 librados: init done 2024-03-25T18:12:55.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.735+0000 7f43f07d0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:55.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.735+0000 7f43f07d0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:55.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.735+0000 7f43f07d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:55.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:55.735+0000 7f43f07d0640 1 -- 172.21.15.37:0/589896605 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43e80b3ef0 con 0x7f43e81430c0 2024-03-25T18:12:56.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.024+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f43e00020e0 con 0x7f43e81430c0 2024-03-25T18:12:56.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.024+0000 7f43deffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:12:56.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.024+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f43e0003b50 con 0x7f43e81430c0 2024-03-25T18:12:56.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.024+0000 7f43deffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:12:56.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.024+0000 7f43deffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:12:56.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.024+0000 7f43deffd640 10 monclient: dump: 2024-03-25T18:12:56.025 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:12:56.026 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:12:56.026 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:56.026 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:12:56.026 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:12:56.026 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:12:56.026 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:12:56.026 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:12:56.026 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:12:56.026 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:12:56.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.024+0000 7f43ecd69640 4 set_mon_vals no callback set 2024-03-25T18:12:56.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.025+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f43e0003d70 con 0x7f43e81430c0 2024-03-25T18:12:56.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.025+0000 7f43ee56c640 10 monclient: get_auth_request con 0x7f43cc06dda0 auth_method 0 2024-03-25T18:12:56.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.026+0000 7f43deffd640 10 client.4459.objecter ms_handle_connect 0x7f43cc06dda0 2024-03-25T18:12:56.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.029+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f43e0054a90 con 0x7f43e81430c0 2024-03-25T18:12:56.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.030+0000 7f43deffd640 10 client.4459.objecter ms_dispatch 0x7f43e8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:12:56.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.030+0000 7f43deffd640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:12:56.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.030+0000 7f43deffd640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:12:56.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.030+0000 7f43deffd640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-25T18:12:56.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.031+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f43e00503d0 con 0x7f43e81430c0 2024-03-25T18:12:56.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.031+0000 7f43deffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:12:56.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.031+0000 7f43deffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:12:56.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.294+0000 7f43f07d0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:56.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.294+0000 7f43f07d0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:56.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.294+0000 7f43f07d0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:12:56.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.294+0000 7f43f07d0640 1 -- 172.21.15.37:0/589896605 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f43e80627b0 con 0x7f43e81430c0 2024-03-25T18:12:56.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.323+0000 7f43deffd640 1 -- 172.21.15.37:0/589896605 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f43e00524d0 con 0x7f43e81430c0 2024-03-25T18:12:56.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.323+0000 7f43deffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:12:56.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.323+0000 7f43deffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:12:56.325 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:12:56.325 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:12:56.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.325+0000 7f43f07d0640 10 librados: watch_flush enter 2024-03-25T18:12:56.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.325+0000 7f43f07d0640 10 librados: watch_flush exit 2024-03-25T18:12:56.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.325+0000 7f43f07d0640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-25T18:12:56.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.325+0000 7f43f07d0640 10 client.4459.objecter successfully canceled tick 2024-03-25T18:12:56.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.325+0000 7f43f07d0640 1 -- 172.21.15.37:0/589896605 >> v1:172.21.15.117:6800/3703993921 conn(0x7f43cc06dda0 legacy=0x7f43cc070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:56.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.325+0000 7f43f07d0640 10 monclient: shutdown 2024-03-25T18:12:56.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.325+0000 7f43f07d0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:12:56.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.325+0000 7f43f07d0640 1 -- 172.21.15.37:0/589896605 >> v1:172.21.15.117:6789/0 conn(0x7f43e81430c0 legacy=0x7f43e81db180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:12:56.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.325+0000 7f43f07d0640 1 -- 172.21.15.37:0/589896605 shutdown_connections 2024-03-25T18:12:56.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.325+0000 7f43f07d0640 1 -- 172.21.15.37:0/589896605 >> 172.21.15.37:0/589896605 conn(0x7f43e80b7ea0 msgr2=0x7f43e8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:12:56.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.326+0000 7f43f07d0640 1 -- 172.21.15.37:0/589896605 shutdown_connections 2024-03-25T18:12:56.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.326+0000 7f43f07d0640 1 -- 172.21.15.37:0/589896605 wait complete. 2024-03-25T18:12:56.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:12:56.326+0000 7f43f07d0640 1 librados: shutdown 2024-03-25T18:12:56.340 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:03.146 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:03.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.266+0000 7fef3cb3b640 10 monclient: get_monmap_and_config 2024-03-25T18:13:03.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.266+0000 7fef3cb3b640 10 monclient: build_initial_monmap 2024-03-25T18:13:03.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.266+0000 7fef3cb3b640 10 monclient: monmap: 2024-03-25T18:13:03.268 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:03.268 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:03.268 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:13:03.267367+0000 2024-03-25T18:13:03.268 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:13:03.267367+0000 2024-03-25T18:13:03.268 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:03.268 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:03.268 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:03.268 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:13:03.269 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:13:03.269 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:03.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.267+0000 7fef3cb3b640 1 Processor -- start 2024-03-25T18:13:03.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.267+0000 7fef3cb3b640 1 -- start start 2024-03-25T18:13:03.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.267+0000 7fef3cb3b640 10 monclient: init 2024-03-25T18:13:03.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.268+0000 7fef3cb3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:03.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.268+0000 7fef3cb3b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:03.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.268+0000 7fef3cb3b640 10 monclient(hunting): picked mon.noname-b con 0x7fef380a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:03.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.268+0000 7fef3cb3b640 10 monclient(hunting): picked mon.noname-a con 0x7fef3813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:03.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.268+0000 7fef3cb3b640 10 monclient(hunting): picked mon.noname-c con 0x7fef381430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:03.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.268+0000 7fef3cb3b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef38142520 con 0x7fef3813f710 2024-03-25T18:13:03.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.268+0000 7fef3cb3b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef380b28a0 con 0x7fef381430c0 2024-03-25T18:13:03.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.268+0000 7fef3cb3b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef380b2a30 con 0x7fef380a98f0 2024-03-25T18:13:03.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.268+0000 7fef3cb3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:03.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.268+0000 7fef3cb3b640 10 monclient(hunting): authenticate will time out at 7138.520445s 2024-03-25T18:13:03.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.271+0000 7fef36d9d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fef381430c0 0x7fef3814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:33002/0 (socket says 172.21.15.37:33002) 2024-03-25T18:13:03.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.271+0000 7fef36d9d640 1 -- 172.21.15.37:0/1982943077 learned_addr learned my addr 172.21.15.37:0/1982943077 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:03.285 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.284+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2589581967 0 0) 0x7fef38142520 con 0x7fef3813f710 2024-03-25T18:13:03.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.284+0000 7fef35d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:13:03.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.284+0000 7fef35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:03.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.284+0000 7fef35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:03.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.284+0000 7fef35d9b640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:13:03.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.284+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef24003520 con 0x7fef3813f710 2024-03-25T18:13:03.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.287+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 816677394 0 0) 0x7fef380b28a0 con 0x7fef381430c0 2024-03-25T18:13:03.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.287+0000 7fef35d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:13:03.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.287+0000 7fef35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:03.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.287+0000 7fef35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:03.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.287+0000 7fef35d9b640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:13:03.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.287+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef38142520 con 0x7fef381430c0 2024-03-25T18:13:03.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.291+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653584736 0 0) 0x7fef380b2a30 con 0x7fef380a98f0 2024-03-25T18:13:03.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.291+0000 7fef35d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:13:03.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.291+0000 7fef35d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:03.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.291+0000 7fef35d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:03.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.291+0000 7fef35d9b640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:13:03.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.291+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef380b28a0 con 0x7fef380a98f0 2024-03-25T18:13:03.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 943516721 0 0) 0x7fef380b28a0 con 0x7fef380a98f0 2024-03-25T18:13:03.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 >> v1:172.21.15.117:6789/0 conn(0x7fef381430c0 legacy=0x7fef3814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:03.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 >> v1:172.21.15.37:6789/0 conn(0x7fef3813f710 legacy=0x7fef381429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:03.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:13:03.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 1 monclient: found mon.noname-b 2024-03-25T18:13:03.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:03.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef380b44c0 con 0x7fef380a98f0 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 10 monclient: _finish_auth 0 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fef280015b0 con 0x7fef380a98f0 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef35d9b640 10 monclient: dump: 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:03.305 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:03.306 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:03.306 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:03.306 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:03.306 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:03.306 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:03.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef3cb3b640 5 monclient: authenticate success, global_id 4457 2024-03-25T18:13:03.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.303+0000 7fef3cb3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:03.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.319+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef28001710 con 0x7fef380a98f0 2024-03-25T18:13:03.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.319+0000 7fef35d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:03.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.319+0000 7fef35d9b640 1 -- 172.21.15.37:0/1982943077 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fef28001b60 con 0x7fef380a98f0 2024-03-25T18:13:03.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.319+0000 7fef35d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:03.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.319+0000 7fef35d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:03.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.319+0000 7fef35d9b640 10 monclient: dump: 2024-03-25T18:13:03.320 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:03.321 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:03.321 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:03.321 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:03.321 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:03.321 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:03.321 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:03.321 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:03.321 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:03.321 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:03.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.319+0000 7fef3cb3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:03.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.319+0000 7fef3cb3b640 4 set_mon_vals no callback set 2024-03-25T18:13:03.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.320+0000 7fef3cb3b640 10 monclient: shutdown 2024-03-25T18:13:03.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.320+0000 7fef3cb3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:03.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.320+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1982943077 >> v1:172.21.15.37:6790/0 conn(0x7fef380a98f0 legacy=0x7fef380a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:03.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.320+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1982943077 shutdown_connections 2024-03-25T18:13:03.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.320+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1982943077 >> 172.21.15.37:0/1982943077 conn(0x7fef380b7ea0 msgr2=0x7fef380b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:03.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.320+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1982943077 shutdown_connections 2024-03-25T18:13:03.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.320+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1982943077 wait complete. 2024-03-25T18:13:03.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.321+0000 7fef3cb3b640 10 monclient: build_initial_monmap 2024-03-25T18:13:03.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.321+0000 7fef3cb3b640 10 monclient: monmap: 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.321+0000 7fef3cb3b640 1 librados: starting msgr at 2024-03-25T18:13:03.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.321+0000 7fef3cb3b640 1 librados: starting objecter 2024-03-25T18:13:03.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.322+0000 7fef3cb3b640 1 Processor -- start 2024-03-25T18:13:03.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.322+0000 7fef3cb3b640 1 -- start start 2024-03-25T18:13:03.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.322+0000 7fef3cb3b640 1 librados: setting wanted keys 2024-03-25T18:13:03.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.322+0000 7fef3cb3b640 1 librados: calling monclient init 2024-03-25T18:13:03.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.322+0000 7fef3cb3b640 10 monclient: init 2024-03-25T18:13:03.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.322+0000 7fef3cb3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:03.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.322+0000 7fef3cb3b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:03.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.322+0000 7fef3cb3b640 10 monclient(hunting): picked mon.noname-a con 0x7fef380a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:03.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.322+0000 7fef3cb3b640 10 monclient(hunting): picked mon.noname-c con 0x7fef3813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:03.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.322+0000 7fef3cb3b640 10 monclient(hunting): picked mon.noname-b con 0x7fef381430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:03.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.322+0000 7fef3cb3b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef381d41e0 con 0x7fef380a98f0 2024-03-25T18:13:03.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.323+0000 7fef3cb3b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef381d44a0 con 0x7fef381430c0 2024-03-25T18:13:03.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.323+0000 7fef3cb3b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef38139a00 con 0x7fef3813f710 2024-03-25T18:13:03.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.323+0000 7fef3cb3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:03.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.323+0000 7fef3cb3b640 10 monclient(hunting): authenticate will time out at 7138.575304s 2024-03-25T18:13:03.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.334+0000 7fef36d9d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fef381430c0 0x7fef381384e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:33036/0 (socket says 172.21.15.37:33036) 2024-03-25T18:13:03.335 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.334+0000 7fef36d9d640 1 -- 172.21.15.37:0/1510378722 learned_addr learned my addr 172.21.15.37:0/1510378722 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:03.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.338+0000 7fef2f7fe640 10 client.?.objecter ms_handle_connect 0x7fef381430c0 2024-03-25T18:13:03.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.338+0000 7fef2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:03.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.343+0000 7fef2f7fe640 10 client.?.objecter ms_handle_connect 0x7fef3813f710 2024-03-25T18:13:03.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.343+0000 7fef2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:03.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.344+0000 7fef2f7fe640 10 client.?.objecter ms_handle_connect 0x7fef380a98f0 2024-03-25T18:13:03.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.344+0000 7fef2f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:03.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.346+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 310858438 0 0) 0x7fef381d44a0 con 0x7fef381430c0 2024-03-25T18:13:03.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.346+0000 7fef2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:03.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.346+0000 7fef2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:03.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.346+0000 7fef2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:03.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.346+0000 7fef2f7fe640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:13:03.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.346+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef18004eb0 con 0x7fef381430c0 2024-03-25T18:13:03.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.352+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 613048500 0 0) 0x7fef381d41e0 con 0x7fef380a98f0 2024-03-25T18:13:03.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.352+0000 7fef2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:03.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.352+0000 7fef2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:03.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.352+0000 7fef2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:03.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.352+0000 7fef2f7fe640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:13:03.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.352+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef381d44a0 con 0x7fef380a98f0 2024-03-25T18:13:03.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.361+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 454845383 0 0) 0x7fef18004eb0 con 0x7fef381430c0 2024-03-25T18:13:03.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.361+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef381d41e0 con 0x7fef381430c0 2024-03-25T18:13:03.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.361+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fef30001c10 con 0x7fef381430c0 2024-03-25T18:13:03.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.361+0000 7fef2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:03.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.361+0000 7fef2f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:03.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.361+0000 7fef2f7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:03.363 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:03.363 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:03.363 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:03.363 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:03.363 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:03.363 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:03.364 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:03.364 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:03.364 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:03.364 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:03.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.361+0000 7fef2f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:03.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.362+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215054136 0 0) 0x7fef38139a00 con 0x7fef3813f710 2024-03-25T18:13:03.364 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.362+0000 7fef2f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:03.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.362+0000 7fef2f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:03.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.362+0000 7fef2f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:03.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.362+0000 7fef2f7fe640 10 monclient(hunting): my global_id is 4460 2024-03-25T18:13:03.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.362+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef18004eb0 con 0x7fef3813f710 2024-03-25T18:13:03.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.378+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3930675264 0 0) 0x7fef18004eb0 con 0x7fef3813f710 2024-03-25T18:13:03.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.378+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fef38139a00 con 0x7fef3813f710 2024-03-25T18:13:03.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.378+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fef200017b0 con 0x7fef3813f710 2024-03-25T18:13:03.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.378+0000 7fef2f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:03.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.378+0000 7fef2f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:03.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.378+0000 7fef2f7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:03.380 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:03.380 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:03.380 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:03.380 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:03.380 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:03.380 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:03.380 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:03.380 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:03.380 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:03.380 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:03.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.380+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4004707848 0 0) 0x7fef381d41e0 con 0x7fef381430c0 2024-03-25T18:13:03.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 >> v1:172.21.15.37:6790/0 conn(0x7fef3813f710 legacy=0x7fef381d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:03.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 >> v1:172.21.15.37:6789/0 conn(0x7fef380a98f0 legacy=0x7fef381d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:03.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef2f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef2f7fe640 1 monclient: found mon.b 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef2f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef2f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef381d7810 con 0x7fef381430c0 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef2f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef2f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef2f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef3cb3b640 5 monclient: authenticate success, global_id 4465 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef3cb3b640 10 monclient: _renew_subs 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef3cb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1510378722 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef381d8440 con 0x7fef381430c0 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef3cb3b640 10 client.4465.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:03.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef3cb3b640 10 monclient: _renew_subs 2024-03-25T18:13:03.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef3cb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:03.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1510378722 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fef381d51e0 con 0x7fef381430c0 2024-03-25T18:13:03.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.381+0000 7fef3cb3b640 1 librados: init done 2024-03-25T18:13:03.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.382+0000 7fef3cb3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:03.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.382+0000 7fef3cb3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:03.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.382+0000 7fef3cb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:03.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.382+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1510378722 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef380627b0 con 0x7fef381430c0 2024-03-25T18:13:03.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.652+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fef30004b60 con 0x7fef381430c0 2024-03-25T18:13:03.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.652+0000 7fef2f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:03.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.652+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fef30004e70 con 0x7fef381430c0 2024-03-25T18:13:03.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.652+0000 7fef2f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.652+0000 7fef2f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.652+0000 7fef2f7fe640 10 monclient: dump: 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:03.654 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:03.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.652+0000 7fef3559a640 4 set_mon_vals no callback set 2024-03-25T18:13:03.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.653+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fef3001ae40 con 0x7fef381430c0 2024-03-25T18:13:03.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.654+0000 7fef3659c640 10 monclient: get_auth_request con 0x7fef1806df70 auth_method 0 2024-03-25T18:13:03.655 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.654+0000 7fef2f7fe640 10 client.4465.objecter ms_handle_connect 0x7fef1806df70 2024-03-25T18:13:03.659 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.658+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fef30054b70 con 0x7fef381430c0 2024-03-25T18:13:03.659 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.658+0000 7fef2f7fe640 10 client.4465.objecter ms_dispatch 0x7fef38000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:03.659 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.658+0000 7fef2f7fe640 3 client.4465.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:03.659 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.658+0000 7fef2f7fe640 3 client.4465.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:03.659 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.658+0000 7fef2f7fe640 20 client.4465.objecter dump_active .. 0 homeless 2024-03-25T18:13:03.659 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.658+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fef300504b0 con 0x7fef381430c0 2024-03-25T18:13:03.659 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.658+0000 7fef2f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:03.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.658+0000 7fef2f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:03.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.912+0000 7fef3cb3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:03.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.912+0000 7fef3cb3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:03.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.912+0000 7fef3cb3b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:03.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.912+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1510378722 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fef38074cc0 con 0x7fef381430c0 2024-03-25T18:13:03.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.943+0000 7fef2f7fe640 1 -- 172.21.15.37:0/1510378722 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fef300525b0 con 0x7fef381430c0 2024-03-25T18:13:03.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.943+0000 7fef2f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:03.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.943+0000 7fef2f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:03.945 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:13:03.945 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:03.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 10 librados: watch_flush enter 2024-03-25T18:13:03.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 10 librados: watch_flush exit 2024-03-25T18:13:03.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 20 client.4465.objecter shutdown clearing up homeless session... 2024-03-25T18:13:03.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 10 client.4465.objecter successfully canceled tick 2024-03-25T18:13:03.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1510378722 >> v1:172.21.15.117:6800/3703993921 conn(0x7fef1806df70 legacy=0x7fef18070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:03.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 10 monclient: shutdown 2024-03-25T18:13:03.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:03.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1510378722 >> v1:172.21.15.117:6789/0 conn(0x7fef381430c0 legacy=0x7fef381384e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:03.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1510378722 shutdown_connections 2024-03-25T18:13:03.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1510378722 >> 172.21.15.37:0/1510378722 conn(0x7fef380b7ea0 msgr2=0x7fef38074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:03.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1510378722 shutdown_connections 2024-03-25T18:13:03.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 1 -- 172.21.15.37:0/1510378722 wait complete. 2024-03-25T18:13:03.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:03.946+0000 7fef3cb3b640 1 librados: shutdown 2024-03-25T18:13:03.959 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:10.072 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:05:45.417 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.131 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:05:45.477 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.168 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:05:45.513 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.188 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:05:45.501 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.192 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:05:45.537 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.193 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:05:45.507 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.201 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:05:45.546 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.214 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:05:45.527 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.227 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:05:45.573 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:10.765 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:10.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.881+0000 7efe5d2ff640 10 monclient: get_monmap_and_config 2024-03-25T18:13:10.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.881+0000 7efe5d2ff640 10 monclient: build_initial_monmap 2024-03-25T18:13:10.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.881+0000 7efe5d2ff640 10 monclient: monmap: 2024-03-25T18:13:10.883 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:10.883 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:10.883 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:13:10.883256+0000 2024-03-25T18:13:10.883 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:13:10.883256+0000 2024-03-25T18:13:10.884 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:10.884 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:10.884 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:10.884 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:13:10.884 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:13:10.884 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:10.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 1 Processor -- start 2024-03-25T18:13:10.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 1 -- start start 2024-03-25T18:13:10.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 10 monclient: init 2024-03-25T18:13:10.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:10.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:10.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 10 monclient(hunting): picked mon.noname-a con 0x7efe580a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:10.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 10 monclient(hunting): picked mon.noname-c con 0x7efe5813f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:10.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 10 monclient(hunting): picked mon.noname-b con 0x7efe581430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:10.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe58142520 con 0x7efe580a98f0 2024-03-25T18:13:10.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe580b28a0 con 0x7efe5813f710 2024-03-25T18:13:10.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe580b2a30 con 0x7efe581430c0 2024-03-25T18:13:10.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:10.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.882+0000 7efe5d2ff640 10 monclient(hunting): authenticate will time out at 7146.136154s 2024-03-25T18:13:10.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.896+0000 7efe567fc640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7efe5813f710 0x7efe581429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:49494/0 (socket says 172.21.15.37:49494) 2024-03-25T18:13:10.898 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.896+0000 7efe567fc640 1 -- 172.21.15.37:0/3947799781 learned_addr learned my addr 172.21.15.37:0/3947799781 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:10.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.907+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2772055382 0 0) 0x7efe580b28a0 con 0x7efe5813f710 2024-03-25T18:13:10.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.907+0000 7efe55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:10.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.907+0000 7efe55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:10.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.907+0000 7efe55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:10.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.907+0000 7efe55ffb640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:13:10.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.907+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe40003520 con 0x7efe5813f710 2024-03-25T18:13:10.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.912+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691103666 0 0) 0x7efe580b2a30 con 0x7efe581430c0 2024-03-25T18:13:10.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.912+0000 7efe55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:10.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.912+0000 7efe55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:10.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.912+0000 7efe55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:10.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.912+0000 7efe55ffb640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:13:10.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.912+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe580b28a0 con 0x7efe581430c0 2024-03-25T18:13:10.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.921+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 746579508 0 0) 0x7efe58142520 con 0x7efe580a98f0 2024-03-25T18:13:10.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.921+0000 7efe55ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:10.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.921+0000 7efe55ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:10.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.921+0000 7efe55ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:10.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.921+0000 7efe55ffb640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:13:10.923 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.921+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe580b2a30 con 0x7efe580a98f0 2024-03-25T18:13:10.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2263431904 0 0) 0x7efe580b28a0 con 0x7efe581430c0 2024-03-25T18:13:10.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 >> v1:172.21.15.117:6789/0 conn(0x7efe5813f710 legacy=0x7efe581429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:10.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 >> v1:172.21.15.37:6789/0 conn(0x7efe580a98f0 legacy=0x7efe580a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:10.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:10.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 1 monclient: found mon.noname-b 2024-03-25T18:13:10.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:10.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe580b44c0 con 0x7efe581430c0 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7efe500019f0 con 0x7efe581430c0 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe55ffb640 10 monclient: dump: 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:10.928 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:10.929 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:10.929 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:10.929 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:10.929 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:10.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe5d2ff640 5 monclient: authenticate success, global_id 4463 2024-03-25T18:13:10.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.925+0000 7efe5d2ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.931+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe50001b50 con 0x7efe581430c0 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.931+0000 7efe55ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.931+0000 7efe55ffb640 1 -- 172.21.15.37:0/3947799781 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7efe50001f60 con 0x7efe581430c0 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.931+0000 7efe55ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.931+0000 7efe55ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.931+0000 7efe55ffb640 10 monclient: dump: 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:10.933 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:10.934 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:10.934 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:10.934 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:10.934 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:10.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.931+0000 7efe5d2ff640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:10.934 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.931+0000 7efe5d2ff640 4 set_mon_vals no callback set 2024-03-25T18:13:10.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.932+0000 7efe5d2ff640 10 monclient: shutdown 2024-03-25T18:13:10.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.932+0000 7efe5d2ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:10.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.932+0000 7efe5d2ff640 1 -- 172.21.15.37:0/3947799781 >> v1:172.21.15.37:6790/0 conn(0x7efe581430c0 legacy=0x7efe5814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:10.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.932+0000 7efe5d2ff640 1 -- 172.21.15.37:0/3947799781 shutdown_connections 2024-03-25T18:13:10.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.932+0000 7efe5d2ff640 1 -- 172.21.15.37:0/3947799781 >> 172.21.15.37:0/3947799781 conn(0x7efe580b7ea0 msgr2=0x7efe580b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:10.935 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.932+0000 7efe5d2ff640 1 -- 172.21.15.37:0/3947799781 shutdown_connections 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.932+0000 7efe5d2ff640 1 -- 172.21.15.37:0/3947799781 wait complete. 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.933+0000 7efe5d2ff640 10 monclient: build_initial_monmap 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.933+0000 7efe5d2ff640 10 monclient: monmap: 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.933+0000 7efe5d2ff640 1 librados: starting msgr at 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.933+0000 7efe5d2ff640 1 librados: starting objecter 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.933+0000 7efe5d2ff640 1 Processor -- start 2024-03-25T18:13:10.936 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.933+0000 7efe5d2ff640 1 -- start start 2024-03-25T18:13:10.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.933+0000 7efe5d2ff640 1 librados: setting wanted keys 2024-03-25T18:13:10.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.933+0000 7efe5d2ff640 1 librados: calling monclient init 2024-03-25T18:13:10.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.933+0000 7efe5d2ff640 10 monclient: init 2024-03-25T18:13:10.937 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.934+0000 7efe5d2ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:10.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.934+0000 7efe5d2ff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:10.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.934+0000 7efe5d2ff640 10 monclient(hunting): picked mon.noname-b con 0x7efe580a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:10.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.934+0000 7efe5d2ff640 10 monclient(hunting): picked mon.noname-c con 0x7efe5813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:10.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.934+0000 7efe5d2ff640 10 monclient(hunting): picked mon.noname-a con 0x7efe581430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:10.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.934+0000 7efe5d2ff640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe581397d0 con 0x7efe581430c0 2024-03-25T18:13:10.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.934+0000 7efe5d2ff640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe5813af80 con 0x7efe580a98f0 2024-03-25T18:13:10.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.934+0000 7efe5d2ff640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe581dba00 con 0x7efe5813f710 2024-03-25T18:13:10.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.934+0000 7efe5d2ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:10.938 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.934+0000 7efe5d2ff640 10 monclient(hunting): authenticate will time out at 7146.187876s 2024-03-25T18:13:10.945 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.943+0000 7efe567fc640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7efe5813f710 0x7efe581dabe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:50904/0 (socket says 172.21.15.37:50904) 2024-03-25T18:13:10.945 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.943+0000 7efe567fc640 1 -- 172.21.15.37:0/1992721798 learned_addr learned my addr 172.21.15.37:0/1992721798 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:10.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.944+0000 7efe3f7fe640 10 client.?.objecter ms_handle_connect 0x7efe5813f710 2024-03-25T18:13:10.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.944+0000 7efe3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:10.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.949+0000 7efe3f7fe640 10 client.?.objecter ms_handle_connect 0x7efe580a98f0 2024-03-25T18:13:10.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.949+0000 7efe3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:10.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.953+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1370578294 0 0) 0x7efe581dba00 con 0x7efe5813f710 2024-03-25T18:13:10.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.953+0000 7efe3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:10.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.953+0000 7efe3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:10.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.953+0000 7efe3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:10.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.953+0000 7efe3f7fe640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:13:10.956 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.953+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe44004eb0 con 0x7efe5813f710 2024-03-25T18:13:10.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.957+0000 7efe3f7fe640 10 client.?.objecter ms_handle_connect 0x7efe581430c0 2024-03-25T18:13:10.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.957+0000 7efe3f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:10.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.960+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2131110898 0 0) 0x7efe5813af80 con 0x7efe580a98f0 2024-03-25T18:13:10.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.960+0000 7efe3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:10.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.960+0000 7efe3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:10.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.960+0000 7efe3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:10.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.960+0000 7efe3f7fe640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:13:10.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.960+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe581dba00 con 0x7efe580a98f0 2024-03-25T18:13:10.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.964+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1091659021 0 0) 0x7efe44004eb0 con 0x7efe5813f710 2024-03-25T18:13:10.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.964+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe5813af80 con 0x7efe5813f710 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.964+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7efe4c001950 con 0x7efe5813f710 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.964+0000 7efe3f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.964+0000 7efe3f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.964+0000 7efe3f7fe640 10 monclient(hunting): dump: 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:10.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.964+0000 7efe3f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:10.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.965+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2311851988 0 0) 0x7efe581397d0 con 0x7efe581430c0 2024-03-25T18:13:10.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.965+0000 7efe3f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:10.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.965+0000 7efe3f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:10.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.965+0000 7efe3f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:10.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.965+0000 7efe3f7fe640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:13:10.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.965+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe44004eb0 con 0x7efe581430c0 2024-03-25T18:13:10.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2233288632 0 0) 0x7efe5813af80 con 0x7efe5813f710 2024-03-25T18:13:10.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 >> v1:172.21.15.117:6789/0 conn(0x7efe580a98f0 legacy=0x7efe580b6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:10.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 >> v1:172.21.15.37:6789/0 conn(0x7efe581430c0 legacy=0x7efe581db2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:10.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe3f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe3f7fe640 1 monclient: found mon.c 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe3f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe3f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe5813b260 con 0x7efe5813f710 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe3f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe3f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe3f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe5d2ff640 5 monclient: authenticate success, global_id 4466 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe5d2ff640 10 monclient: _renew_subs 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe5d2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe5d2ff640 1 -- 172.21.15.37:0/1992721798 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe580b74f0 con 0x7efe5813f710 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe5d2ff640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe5d2ff640 10 monclient: _renew_subs 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe5d2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe5d2ff640 1 -- 172.21.15.37:0/1992721798 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5812f7e0 con 0x7efe5813f710 2024-03-25T18:13:10.976 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.973+0000 7efe5d2ff640 1 librados: init done 2024-03-25T18:13:10.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.974+0000 7efe5d2ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:10.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.974+0000 7efe5d2ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:10.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.974+0000 7efe5d2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:10.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:10.974+0000 7efe5d2ff640 1 -- 172.21.15.37:0/1992721798 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe580627b0 con 0x7efe5813f710 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.292+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe4c0047e0 con 0x7efe5813f710 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.292+0000 7efe3f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.292+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7efe4c001d80 con 0x7efe5813f710 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.292+0000 7efe3f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.292+0000 7efe3f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.292+0000 7efe3f7fe640 10 monclient: dump: 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:11.295 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:11.296 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:11.296 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:11.296 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.292+0000 7efe557fa640 4 set_mon_vals no callback set 2024-03-25T18:13:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.293+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7efe4c004c30 con 0x7efe5813f710 2024-03-25T18:13:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.294+0000 7efe56ffd640 10 monclient: get_auth_request con 0x7efe4406de00 auth_method 0 2024-03-25T18:13:11.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.294+0000 7efe3f7fe640 10 client.4466.objecter ms_handle_connect 0x7efe4406de00 2024-03-25T18:13:11.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.302+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7efe4c054810 con 0x7efe5813f710 2024-03-25T18:13:11.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.302+0000 7efe3f7fe640 10 client.4466.objecter ms_dispatch 0x7efe58000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:11.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.302+0000 7efe3f7fe640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:11.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.302+0000 7efe3f7fe640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:11.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.302+0000 7efe3f7fe640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-25T18:13:11.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.302+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe4c050150 con 0x7efe5813f710 2024-03-25T18:13:11.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.302+0000 7efe3f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:11.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.302+0000 7efe3f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:11.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.546+0000 7efe5d2ff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:11.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.546+0000 7efe5d2ff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:11.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.546+0000 7efe5d2ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:11.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.546+0000 7efe5d2ff640 1 -- 172.21.15.37:0/1992721798 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe58139980 con 0x7efe5813f710 2024-03-25T18:13:11.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.583+0000 7efe3f7fe640 1 -- 172.21.15.37:0/1992721798 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7efe58139980 con 0x7efe5813f710 2024-03-25T18:13:11.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.583+0000 7efe3f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:11.585 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.583+0000 7efe3f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:11.585 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:13:11.585 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:11.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.585+0000 7efe5d2ff640 10 librados: watch_flush enter 2024-03-25T18:13:11.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.585+0000 7efe5d2ff640 10 librados: watch_flush exit 2024-03-25T18:13:11.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.585+0000 7efe5d2ff640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-25T18:13:11.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.585+0000 7efe5d2ff640 10 client.4466.objecter successfully canceled tick 2024-03-25T18:13:11.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.586+0000 7efe5d2ff640 1 -- 172.21.15.37:0/1992721798 >> v1:172.21.15.117:6800/3703993921 conn(0x7efe4406de00 legacy=0x7efe440702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:11.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.586+0000 7efe5d2ff640 10 monclient: shutdown 2024-03-25T18:13:11.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.586+0000 7efe5d2ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:11.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.586+0000 7efe5d2ff640 1 -- 172.21.15.37:0/1992721798 >> v1:172.21.15.37:6790/0 conn(0x7efe5813f710 legacy=0x7efe581dabe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:11.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.586+0000 7efe5d2ff640 1 -- 172.21.15.37:0/1992721798 shutdown_connections 2024-03-25T18:13:11.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.586+0000 7efe5d2ff640 1 -- 172.21.15.37:0/1992721798 >> 172.21.15.37:0/1992721798 conn(0x7efe580b7ea0 msgr2=0x7efe58074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:11.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.586+0000 7efe5d2ff640 1 -- 172.21.15.37:0/1992721798 shutdown_connections 2024-03-25T18:13:11.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.586+0000 7efe5d2ff640 1 -- 172.21.15.37:0/1992721798 wait complete. 2024-03-25T18:13:11.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:11.586+0000 7efe5d2ff640 1 librados: shutdown 2024-03-25T18:13:11.599 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:18.406 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:18.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.519+0000 7fa6a75ed640 10 monclient: get_monmap_and_config 2024-03-25T18:13:18.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.519+0000 7fa6a75ed640 10 monclient: build_initial_monmap 2024-03-25T18:13:18.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.519+0000 7fa6a75ed640 10 monclient: monmap: 2024-03-25T18:13:18.520 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:18.520 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:18.520 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:13:18.520407+0000 2024-03-25T18:13:18.520 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:13:18.520407+0000 2024-03-25T18:13:18.520 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:18.521 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:18.521 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:18.521 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:13:18.521 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:13:18.521 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:18.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.520+0000 7fa6a75ed640 1 Processor -- start 2024-03-25T18:13:18.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.520+0000 7fa6a75ed640 1 -- start start 2024-03-25T18:13:18.521 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.520+0000 7fa6a75ed640 10 monclient: init 2024-03-25T18:13:18.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.521+0000 7fa6a75ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:18.522 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.521+0000 7fa6a75ed640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:18.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.521+0000 7fa6a75ed640 10 monclient(hunting): picked mon.noname-a con 0x7fa6a00a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:18.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.521+0000 7fa6a75ed640 10 monclient(hunting): picked mon.noname-c con 0x7fa6a013f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:18.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.521+0000 7fa6a75ed640 10 monclient(hunting): picked mon.noname-b con 0x7fa6a01430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:18.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.521+0000 7fa6a75ed640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a0142520 con 0x7fa6a00a98f0 2024-03-25T18:13:18.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.521+0000 7fa6a75ed640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a00b28a0 con 0x7fa6a013f710 2024-03-25T18:13:18.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.521+0000 7fa6a75ed640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a00b2a30 con 0x7fa6a01430c0 2024-03-25T18:13:18.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.521+0000 7fa6a75ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:18.523 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.521+0000 7fa6a75ed640 10 monclient(hunting): authenticate will time out at 7153.774077s 2024-03-25T18:13:18.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.525+0000 7fa6a5b8a640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fa6a01430c0 0x7fa6a014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:34294/0 (socket says 172.21.15.37:34294) 2024-03-25T18:13:18.526 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.525+0000 7fa6a5b8a640 1 -- 172.21.15.37:0/2710522848 learned_addr learned my addr 172.21.15.37:0/2710522848 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:18.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.544+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164926917 0 0) 0x7fa6a00b2a30 con 0x7fa6a01430c0 2024-03-25T18:13:18.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.544+0000 7fa697fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:18.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.544+0000 7fa697fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:18.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.544+0000 7fa697fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:18.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.544+0000 7fa697fff640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:13:18.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.544+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa68c003520 con 0x7fa6a01430c0 2024-03-25T18:13:18.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.549+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2859273461 0 0) 0x7fa6a0142520 con 0x7fa6a00a98f0 2024-03-25T18:13:18.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.549+0000 7fa697fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:18.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.549+0000 7fa697fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:18.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.549+0000 7fa697fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:18.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.549+0000 7fa697fff640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:13:18.551 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.549+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a00b2a30 con 0x7fa6a00a98f0 2024-03-25T18:13:18.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.556+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4224383402 0 0) 0x7fa6a00b28a0 con 0x7fa6a013f710 2024-03-25T18:13:18.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.556+0000 7fa697fff640 10 monclient(hunting): _negotiate 2024-03-25T18:13:18.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.556+0000 7fa697fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:18.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.556+0000 7fa697fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:18.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.556+0000 7fa697fff640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:13:18.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.556+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a0142520 con 0x7fa6a013f710 2024-03-25T18:13:18.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1659751616 0 0) 0x7fa68c003520 con 0x7fa6a01430c0 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 >> v1:172.21.15.117:6789/0 conn(0x7fa6a013f710 legacy=0x7fa6a01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 >> v1:172.21.15.37:6789/0 conn(0x7fa6a00a98f0 legacy=0x7fa6a00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 10 monclient: _finish_hunting 0 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 1 monclient: found mon.noname-b 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a00b44c0 con 0x7fa6a01430c0 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 10 monclient: _finish_auth 0 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 10 monclient: _check_auth_tickets 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa698001a10 con 0x7fa6a01430c0 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:18.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa697fff640 10 monclient: dump: 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa6a75ed640 5 monclient: authenticate success, global_id 4469 2024-03-25T18:13:18.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.562+0000 7fa6a75ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:18.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.587+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa698001c30 con 0x7fa6a01430c0 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.587+0000 7fa697fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.587+0000 7fa697fff640 1 -- 172.21.15.37:0/2710522848 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa698001fc0 con 0x7fa6a01430c0 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.587+0000 7fa6a75ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.587+0000 7fa6a75ed640 4 set_mon_vals no callback set 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.587+0000 7fa697fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.587+0000 7fa697fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.587+0000 7fa697fff640 10 monclient: dump: 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:18.589 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:18.590 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:18.590 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:18.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.588+0000 7fa6a75ed640 10 monclient: shutdown 2024-03-25T18:13:18.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.588+0000 7fa6a75ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:18.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.588+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2710522848 >> v1:172.21.15.37:6790/0 conn(0x7fa6a01430c0 legacy=0x7fa6a014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:18.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.588+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2710522848 shutdown_connections 2024-03-25T18:13:18.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.588+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2710522848 >> 172.21.15.37:0/2710522848 conn(0x7fa6a00b7ea0 msgr2=0x7fa6a00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:18.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.588+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2710522848 shutdown_connections 2024-03-25T18:13:18.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.588+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2710522848 wait complete. 2024-03-25T18:13:18.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.589+0000 7fa6a75ed640 10 monclient: build_initial_monmap 2024-03-25T18:13:18.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.589+0000 7fa6a75ed640 10 monclient: monmap: 2024-03-25T18:13:18.592 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:18.592 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:18.592 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:13:18.592 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:13:18.592 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:18.592 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:13:18.592 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:18.592 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:13:18.592 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:13:18.593 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:18.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.590+0000 7fa6a75ed640 1 librados: starting msgr at 2024-03-25T18:13:18.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.590+0000 7fa6a75ed640 1 librados: starting objecter 2024-03-25T18:13:18.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.590+0000 7fa6a75ed640 1 Processor -- start 2024-03-25T18:13:18.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.590+0000 7fa6a75ed640 1 -- start start 2024-03-25T18:13:18.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.590+0000 7fa6a75ed640 1 librados: setting wanted keys 2024-03-25T18:13:18.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.590+0000 7fa6a75ed640 1 librados: calling monclient init 2024-03-25T18:13:18.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.590+0000 7fa6a75ed640 10 monclient: init 2024-03-25T18:13:18.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.590+0000 7fa6a75ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:18.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.590+0000 7fa6a75ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:18.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.590+0000 7fa6a75ed640 10 monclient(hunting): picked mon.noname-a con 0x7fa6a00a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:18.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.591+0000 7fa6a75ed640 10 monclient(hunting): picked mon.noname-c con 0x7fa6a013f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:18.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.591+0000 7fa6a75ed640 10 monclient(hunting): picked mon.noname-b con 0x7fa6a01430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:18.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.591+0000 7fa6a75ed640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a01396e0 con 0x7fa6a00a98f0 2024-03-25T18:13:18.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.591+0000 7fa6a75ed640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a0138530 con 0x7fa6a01430c0 2024-03-25T18:13:18.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.591+0000 7fa6a75ed640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6a01399c0 con 0x7fa6a013f710 2024-03-25T18:13:18.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.591+0000 7fa6a75ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:18.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.591+0000 7fa6a75ed640 10 monclient(hunting): authenticate will time out at 7153.843483s 2024-03-25T18:13:18.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.600+0000 7fa6a5389640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fa6a00a98f0 0x7fa6a0137820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:60732/0 (socket says 172.21.15.37:60732) 2024-03-25T18:13:18.600 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.600+0000 7fa6a5389640 1 -- 172.21.15.37:0/2891387324 learned_addr learned my addr 172.21.15.37:0/2891387324 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:18.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.602+0000 7fa695ffb640 10 client.?.objecter ms_handle_connect 0x7fa6a00a98f0 2024-03-25T18:13:18.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.602+0000 7fa695ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:18.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.603+0000 7fa695ffb640 10 client.?.objecter ms_handle_connect 0x7fa6a013f710 2024-03-25T18:13:18.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.603+0000 7fa695ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:18.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.607+0000 7fa695ffb640 10 client.?.objecter ms_handle_connect 0x7fa6a01430c0 2024-03-25T18:13:18.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.607+0000 7fa695ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:18.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 922619785 0 0) 0x7fa6a01399c0 con 0x7fa6a013f710 2024-03-25T18:13:18.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:18.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:18.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:18.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:13:18.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa684004eb0 con 0x7fa6a013f710 2024-03-25T18:13:18.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569940038 0 0) 0x7fa6a0138530 con 0x7fa6a01430c0 2024-03-25T18:13:18.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:18.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:18.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:18.620 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:13:18.621 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.619+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a01399c0 con 0x7fa6a01430c0 2024-03-25T18:13:18.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.628+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2163276038 0 0) 0x7fa6a01396e0 con 0x7fa6a00a98f0 2024-03-25T18:13:18.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.628+0000 7fa695ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:13:18.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.628+0000 7fa695ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:18.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.628+0000 7fa695ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:18.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.628+0000 7fa695ffb640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:13:18.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.628+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa6a0138530 con 0x7fa6a00a98f0 2024-03-25T18:13:18.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.633+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2548409791 0 0) 0x7fa684004eb0 con 0x7fa6a013f710 2024-03-25T18:13:18.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.633+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa6a01396e0 con 0x7fa6a013f710 2024-03-25T18:13:18.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.633+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa6900018f0 con 0x7fa6a013f710 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.633+0000 7fa695ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.633+0000 7fa695ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.633+0000 7fa695ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:18.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.633+0000 7fa695ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:18.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.639+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4013482147 0 0) 0x7fa6a01399c0 con 0x7fa6a01430c0 2024-03-25T18:13:18.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.639+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa684004eb0 con 0x7fa6a01430c0 2024-03-25T18:13:18.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.639+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa698001da0 con 0x7fa6a01430c0 2024-03-25T18:13:18.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.639+0000 7fa695ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:18.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.639+0000 7fa695ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:18.640 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.639+0000 7fa695ffb640 10 monclient(hunting): dump: 2024-03-25T18:13:18.640 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:18.640 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:18.640 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:18.640 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:18.640 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:18.641 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:18.641 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:18.641 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:18.641 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:18.641 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3486933667 0 0) 0x7fa6a01396e0 con 0x7fa6a013f710 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 >> v1:172.21.15.117:6789/0 conn(0x7fa6a01430c0 legacy=0x7fa6a01e1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 >> v1:172.21.15.37:6789/0 conn(0x7fa6a00a98f0 legacy=0x7fa6a0137820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa695ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa695ffb640 1 monclient: found mon.c 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa695ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa695ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a0138a50 con 0x7fa6a013f710 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa695ffb640 10 monclient: _finish_auth 0 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa695ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa695ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:18.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa6a75ed640 5 monclient: authenticate success, global_id 4472 2024-03-25T18:13:18.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa6a75ed640 10 monclient: _renew_subs 2024-03-25T18:13:18.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa6a75ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:18.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2891387324 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6a01380a0 con 0x7fa6a013f710 2024-03-25T18:13:18.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa6a75ed640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:18.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa6a75ed640 10 monclient: _renew_subs 2024-03-25T18:13:18.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa6a75ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:18.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2891387324 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6a01db140 con 0x7fa6a013f710 2024-03-25T18:13:18.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.645+0000 7fa6a75ed640 1 librados: init done 2024-03-25T18:13:18.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.646+0000 7fa6a75ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:18.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.646+0000 7fa6a75ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:18.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.646+0000 7fa6a75ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:18.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.646+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2891387324 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6a0138ca0 con 0x7fa6a013f710 2024-03-25T18:13:18.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.951+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa690003610 con 0x7fa6a013f710 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.951+0000 7fa695ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.951+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa690003780 con 0x7fa6a013f710 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.951+0000 7fa695ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.951+0000 7fa695ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.951+0000 7fa695ffb640 10 monclient: dump: 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:18.953 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:18.954 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:18.954 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:18.954 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:18.954 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.951+0000 7fa6977fe640 4 set_mon_vals no callback set 2024-03-25T18:13:18.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.952+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fa690019750 con 0x7fa6a013f710 2024-03-25T18:13:18.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.953+0000 7fa6a5389640 10 monclient: get_auth_request con 0x7fa68406df70 auth_method 0 2024-03-25T18:13:18.955 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.953+0000 7fa695ffb640 10 client.4472.objecter ms_handle_connect 0x7fa68406df70 2024-03-25T18:13:18.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.959+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fa690053450 con 0x7fa6a013f710 2024-03-25T18:13:18.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.959+0000 7fa695ffb640 10 client.4472.objecter ms_dispatch 0x7fa6a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:18.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.959+0000 7fa695ffb640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:18.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.959+0000 7fa695ffb640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:18.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.959+0000 7fa695ffb640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-25T18:13:18.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.959+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa69004ed90 con 0x7fa6a013f710 2024-03-25T18:13:18.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.959+0000 7fa695ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:18.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:18.959+0000 7fa695ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:19.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.221+0000 7fa6a75ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:19.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.221+0000 7fa6a75ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:19.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.221+0000 7fa6a75ed640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:19.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.221+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2891387324 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa6a01386c0 con 0x7fa6a013f710 2024-03-25T18:13:19.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.256+0000 7fa695ffb640 1 -- 172.21.15.37:0/2891387324 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fa690050e90 con 0x7fa6a013f710 2024-03-25T18:13:19.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.256+0000 7fa695ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:19.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.256+0000 7fa695ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:19.258 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:13:19.258 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:19.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 10 librados: watch_flush enter 2024-03-25T18:13:19.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 10 librados: watch_flush exit 2024-03-25T18:13:19.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-25T18:13:19.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 10 client.4472.objecter successfully canceled tick 2024-03-25T18:13:19.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2891387324 >> v1:172.21.15.117:6800/3703993921 conn(0x7fa68406df70 legacy=0x7fa684070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:19.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 10 monclient: shutdown 2024-03-25T18:13:19.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:19.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2891387324 >> v1:172.21.15.37:6790/0 conn(0x7fa6a013f710 legacy=0x7fa6a01da850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:19.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2891387324 shutdown_connections 2024-03-25T18:13:19.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2891387324 >> 172.21.15.37:0/2891387324 conn(0x7fa6a00b7ea0 msgr2=0x7fa6a0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:19.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2891387324 shutdown_connections 2024-03-25T18:13:19.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 1 -- 172.21.15.37:0/2891387324 wait complete. 2024-03-25T18:13:19.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:19.259+0000 7fa6a75ed640 1 librados: shutdown 2024-03-25T18:13:19.272 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:26.079 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:26.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.193+0000 7fd3f58f6640 10 monclient: get_monmap_and_config 2024-03-25T18:13:26.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.193+0000 7fd3f58f6640 10 monclient: build_initial_monmap 2024-03-25T18:13:26.194 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.193+0000 7fd3f58f6640 10 monclient: monmap: 2024-03-25T18:13:26.195 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:26.195 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:26.195 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:13:26.194564+0000 2024-03-25T18:13:26.195 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:13:26.194564+0000 2024-03-25T18:13:26.195 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:26.195 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:26.195 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:26.195 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:13:26.195 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:13:26.195 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:26.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.194+0000 7fd3f58f6640 1 Processor -- start 2024-03-25T18:13:26.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.194+0000 7fd3f58f6640 1 -- start start 2024-03-25T18:13:26.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.194+0000 7fd3f58f6640 10 monclient: init 2024-03-25T18:13:26.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.195+0000 7fd3f58f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:26.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.195+0000 7fd3f58f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:26.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.195+0000 7fd3f58f6640 10 monclient(hunting): picked mon.noname-a con 0x7fd3f00a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:26.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.195+0000 7fd3f58f6640 10 monclient(hunting): picked mon.noname-b con 0x7fd3f013f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:26.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.195+0000 7fd3f58f6640 10 monclient(hunting): picked mon.noname-c con 0x7fd3f01430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:26.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.195+0000 7fd3f58f6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f0142520 con 0x7fd3f00a98f0 2024-03-25T18:13:26.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.195+0000 7fd3f58f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f00b28a0 con 0x7fd3f01430c0 2024-03-25T18:13:26.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.195+0000 7fd3f58f6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f00b2a30 con 0x7fd3f013f710 2024-03-25T18:13:26.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.195+0000 7fd3f58f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:26.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.195+0000 7fd3f58f6640 10 monclient(hunting): authenticate will time out at 7161.447753s 2024-03-25T18:13:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.199+0000 7fd3eeffd640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7fd3f00a98f0 0x7fd3f00a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:36896/0 (socket says 172.21.15.37:36896) 2024-03-25T18:13:26.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.199+0000 7fd3eeffd640 1 -- 172.21.15.37:0/2374834457 learned_addr learned my addr 172.21.15.37:0/2374834457 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:26.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.204+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781079367 0 0) 0x7fd3f0142520 con 0x7fd3f00a98f0 2024-03-25T18:13:26.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.204+0000 7fd3ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:26.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.204+0000 7fd3ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:26.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.204+0000 7fd3ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:26.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.204+0000 7fd3ee7fc640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:13:26.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.204+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3dc003520 con 0x7fd3f00a98f0 2024-03-25T18:13:26.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.210+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2295592157 0 0) 0x7fd3f00b28a0 con 0x7fd3f01430c0 2024-03-25T18:13:26.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.210+0000 7fd3ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:26.211 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.210+0000 7fd3ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:26.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.210+0000 7fd3ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:26.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.210+0000 7fd3ee7fc640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:13:26.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.210+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f0142520 con 0x7fd3f01430c0 2024-03-25T18:13:26.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.211+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2892618311 0 0) 0x7fd3f00b2a30 con 0x7fd3f013f710 2024-03-25T18:13:26.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.211+0000 7fd3ee7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:13:26.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.211+0000 7fd3ee7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:26.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.211+0000 7fd3ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:26.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.211+0000 7fd3ee7fc640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:13:26.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.211+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f00b28a0 con 0x7fd3f013f710 2024-03-25T18:13:26.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 251550773 0 0) 0x7fd3dc003520 con 0x7fd3f00a98f0 2024-03-25T18:13:26.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 >> v1:172.21.15.37:6790/0 conn(0x7fd3f013f710 legacy=0x7fd3f01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:26.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 >> v1:172.21.15.117:6789/0 conn(0x7fd3f01430c0 legacy=0x7fd3f014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:26.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:13:26.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 1 monclient: found mon.noname-a 2024-03-25T18:13:26.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:26.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:13:26.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f00b44c0 con 0x7fd3f00a98f0 2024-03-25T18:13:26.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 10 monclient: _finish_auth 0 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd3e0001690 con 0x7fd3f00a98f0 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3ee7fc640 10 monclient: dump: 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:26.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3f58f6640 5 monclient: authenticate success, global_id 4479 2024-03-25T18:13:26.218 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.214+0000 7fd3f58f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:26.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.235+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3e00017f0 con 0x7fd3f00a98f0 2024-03-25T18:13:26.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.235+0000 7fd3ee7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:26.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.235+0000 7fd3ee7fc640 1 -- 172.21.15.37:0/2374834457 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd3e0001c00 con 0x7fd3f00a98f0 2024-03-25T18:13:26.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.235+0000 7fd3f58f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:26.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.235+0000 7fd3f58f6640 4 set_mon_vals no callback set 2024-03-25T18:13:26.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.235+0000 7fd3ee7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:26.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.235+0000 7fd3ee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:26.236 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.235+0000 7fd3ee7fc640 10 monclient: dump: 2024-03-25T18:13:26.236 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:26.236 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:26.236 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:26.237 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:26.237 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:26.237 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:26.237 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:26.237 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:26.237 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:26.237 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:26.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.235+0000 7fd3f58f6640 10 monclient: shutdown 2024-03-25T18:13:26.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.235+0000 7fd3f58f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:26.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.235+0000 7fd3f58f6640 1 -- 172.21.15.37:0/2374834457 >> v1:172.21.15.37:6789/0 conn(0x7fd3f00a98f0 legacy=0x7fd3f00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:26.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.236+0000 7fd3f58f6640 1 -- 172.21.15.37:0/2374834457 shutdown_connections 2024-03-25T18:13:26.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.236+0000 7fd3f58f6640 1 -- 172.21.15.37:0/2374834457 >> 172.21.15.37:0/2374834457 conn(0x7fd3f00b7ea0 msgr2=0x7fd3f00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:26.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.236+0000 7fd3f58f6640 1 -- 172.21.15.37:0/2374834457 shutdown_connections 2024-03-25T18:13:26.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.236+0000 7fd3f58f6640 1 -- 172.21.15.37:0/2374834457 wait complete. 2024-03-25T18:13:26.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.236+0000 7fd3f58f6640 10 monclient: build_initial_monmap 2024-03-25T18:13:26.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.236+0000 7fd3f58f6640 10 monclient: monmap: 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.237+0000 7fd3f58f6640 1 librados: starting msgr at 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.237+0000 7fd3f58f6640 1 librados: starting objecter 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.237+0000 7fd3f58f6640 1 Processor -- start 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.237+0000 7fd3f58f6640 1 -- start start 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.237+0000 7fd3f58f6640 1 librados: setting wanted keys 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.237+0000 7fd3f58f6640 1 librados: calling monclient init 2024-03-25T18:13:26.240 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.237+0000 7fd3f58f6640 10 monclient: init 2024-03-25T18:13:26.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.238+0000 7fd3f58f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:26.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.238+0000 7fd3f58f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:26.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.238+0000 7fd3f58f6640 10 monclient(hunting): picked mon.noname-a con 0x7fd3f00a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:26.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.238+0000 7fd3f58f6640 10 monclient(hunting): picked mon.noname-b con 0x7fd3f013f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:26.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.238+0000 7fd3f58f6640 10 monclient(hunting): picked mon.noname-c con 0x7fd3f01430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:26.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.238+0000 7fd3f58f6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f00abd30 con 0x7fd3f00a98f0 2024-03-25T18:13:26.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.238+0000 7fd3f58f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f01d77a0 con 0x7fd3f013f710 2024-03-25T18:13:26.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.238+0000 7fd3f58f6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3f01d51f0 con 0x7fd3f01430c0 2024-03-25T18:13:26.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.238+0000 7fd3f58f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:26.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.238+0000 7fd3f58f6640 10 monclient(hunting): authenticate will time out at 7161.490483s 2024-03-25T18:13:26.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.245+0000 7fd3ef7fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fd3f01430c0 0x7fd3f01e1ea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:38966/0 (socket says 172.21.15.37:38966) 2024-03-25T18:13:26.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.245+0000 7fd3ef7fe640 1 -- 172.21.15.37:0/1138469426 learned_addr learned my addr 172.21.15.37:0/1138469426 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:26.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.246+0000 7fd3ecff9640 10 client.?.objecter ms_handle_connect 0x7fd3f01430c0 2024-03-25T18:13:26.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.246+0000 7fd3ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:26.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.248+0000 7fd3ecff9640 10 client.?.objecter ms_handle_connect 0x7fd3f013f710 2024-03-25T18:13:26.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.248+0000 7fd3ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:26.254 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.252+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1479111960 0 0) 0x7fd3f01d51f0 con 0x7fd3f01430c0 2024-03-25T18:13:26.254 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.252+0000 7fd3ecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:26.254 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.252+0000 7fd3ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:26.254 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.252+0000 7fd3ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:26.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.252+0000 7fd3ecff9640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:13:26.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.252+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3d0004eb0 con 0x7fd3f01430c0 2024-03-25T18:13:26.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.256+0000 7fd3ecff9640 10 client.?.objecter ms_handle_connect 0x7fd3f00a98f0 2024-03-25T18:13:26.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.256+0000 7fd3ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:26.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.258+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372775955 0 0) 0x7fd3f01d77a0 con 0x7fd3f013f710 2024-03-25T18:13:26.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.258+0000 7fd3ecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:26.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.258+0000 7fd3ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:26.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.258+0000 7fd3ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:26.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.258+0000 7fd3ecff9640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:13:26.260 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.258+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f01d51f0 con 0x7fd3f013f710 2024-03-25T18:13:26.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.268+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2309278912 0 0) 0x7fd3f01d51f0 con 0x7fd3f013f710 2024-03-25T18:13:26.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.268+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd3f01d77a0 con 0x7fd3f013f710 2024-03-25T18:13:26.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.268+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd3d8001930 con 0x7fd3f013f710 2024-03-25T18:13:26.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.268+0000 7fd3ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:26.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.268+0000 7fd3ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:26.269 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.268+0000 7fd3ecff9640 10 monclient(hunting): dump: 2024-03-25T18:13:26.270 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:26.270 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:26.270 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:26.270 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:26.270 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:26.270 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:26.270 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:26.270 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:26.270 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:26.270 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:26.270 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.268+0000 7fd3ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:26.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.270+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123884572 0 0) 0x7fd3f00abd30 con 0x7fd3f00a98f0 2024-03-25T18:13:26.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.270+0000 7fd3ecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:26.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.270+0000 7fd3ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:26.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.270+0000 7fd3ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:26.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.270+0000 7fd3ecff9640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:13:26.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.270+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd3f01d51f0 con 0x7fd3f00a98f0 2024-03-25T18:13:26.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.278+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2014849762 0 0) 0x7fd3f01d77a0 con 0x7fd3f013f710 2024-03-25T18:13:26.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.278+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 >> v1:172.21.15.37:6790/0 conn(0x7fd3f01430c0 legacy=0x7fd3f01e1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:26.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.278+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 >> v1:172.21.15.37:6789/0 conn(0x7fd3f00a98f0 legacy=0x7fd3f01d7cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:26.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.278+0000 7fd3ecff9640 10 monclient: _finish_hunting 0 2024-03-25T18:13:26.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.278+0000 7fd3ecff9640 1 monclient: found mon.b 2024-03-25T18:13:26.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.278+0000 7fd3ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.278+0000 7fd3ecff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.278+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3f01d83d0 con 0x7fd3f013f710 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.278+0000 7fd3ecff9640 10 monclient: _finish_auth 0 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.278+0000 7fd3ecff9640 10 monclient: _check_auth_tickets 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.278+0000 7fd3ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.279+0000 7fd3f58f6640 5 monclient: authenticate success, global_id 4483 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.279+0000 7fd3f58f6640 10 monclient: _renew_subs 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.279+0000 7fd3f58f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.279+0000 7fd3f58f6640 1 -- 172.21.15.37:0/1138469426 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3f0138a50 con 0x7fd3f013f710 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.279+0000 7fd3f58f6640 10 client.4483.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.279+0000 7fd3f58f6640 10 monclient: _renew_subs 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.279+0000 7fd3f58f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.279+0000 7fd3f58f6640 1 -- 172.21.15.37:0/1138469426 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3f01e35c0 con 0x7fd3f013f710 2024-03-25T18:13:26.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.279+0000 7fd3f58f6640 1 librados: init done 2024-03-25T18:13:26.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.280+0000 7fd3f58f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:26.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.280+0000 7fd3f58f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:26.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.280+0000 7fd3f58f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:26.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.280+0000 7fd3f58f6640 1 -- 172.21.15.37:0/1138469426 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3f00627b0 con 0x7fd3f013f710 2024-03-25T18:13:26.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.560+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd3d8003610 con 0x7fd3f013f710 2024-03-25T18:13:26.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.560+0000 7fd3ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:26.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.560+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd3d8001610 con 0x7fd3f013f710 2024-03-25T18:13:26.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.560+0000 7fd3ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:26.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.560+0000 7fd3ecff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:26.562 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.560+0000 7fd3ecff9640 10 monclient: dump: 2024-03-25T18:13:26.562 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:26.562 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:26.562 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:26.562 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:26.563 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:26.563 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:26.563 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:26.563 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:26.563 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:26.563 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:26.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.560+0000 7fd3edffb640 4 set_mon_vals no callback set 2024-03-25T18:13:26.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.561+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fd3d8003b20 con 0x7fd3f013f710 2024-03-25T18:13:26.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.562+0000 7fd3eeffd640 10 monclient: get_auth_request con 0x7fd3d006de50 auth_method 0 2024-03-25T18:13:26.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.562+0000 7fd3ecff9640 10 client.4483.objecter ms_handle_connect 0x7fd3d006de50 2024-03-25T18:13:26.568 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.567+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fd3d8053600 con 0x7fd3f013f710 2024-03-25T18:13:26.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.567+0000 7fd3ecff9640 10 client.4483.objecter ms_dispatch 0x7fd3f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:26.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.567+0000 7fd3ecff9640 3 client.4483.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:26.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.567+0000 7fd3ecff9640 3 client.4483.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:26.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.567+0000 7fd3ecff9640 20 client.4483.objecter dump_active .. 0 homeless 2024-03-25T18:13:26.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.568+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd3d804f110 con 0x7fd3f013f710 2024-03-25T18:13:26.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.568+0000 7fd3ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:26.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.568+0000 7fd3ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:26.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.792+0000 7fd3f58f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:26.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.792+0000 7fd3f58f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:26.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.792+0000 7fd3f58f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:26.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.792+0000 7fd3f58f6640 1 -- 172.21.15.37:0/1138469426 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3f013e250 con 0x7fd3f013f710 2024-03-25T18:13:26.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.820+0000 7fd3ecff9640 1 -- 172.21.15.37:0/1138469426 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fd3d8051040 con 0x7fd3f013f710 2024-03-25T18:13:26.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.820+0000 7fd3ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:26.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.820+0000 7fd3ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:26.822 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:13:26.822 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 10 librados: watch_flush enter 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 10 librados: watch_flush exit 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 20 client.4483.objecter shutdown clearing up homeless session... 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 10 client.4483.objecter successfully canceled tick 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 1 -- 172.21.15.37:0/1138469426 >> v1:172.21.15.117:6800/3703993921 conn(0x7fd3d006de50 legacy=0x7fd3d0070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 10 monclient: shutdown 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 1 -- 172.21.15.37:0/1138469426 >> v1:172.21.15.117:6789/0 conn(0x7fd3f013f710 legacy=0x7fd3f0138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 1 -- 172.21.15.37:0/1138469426 shutdown_connections 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 1 -- 172.21.15.37:0/1138469426 >> 172.21.15.37:0/1138469426 conn(0x7fd3f00b7ea0 msgr2=0x7fd3f0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 1 -- 172.21.15.37:0/1138469426 shutdown_connections 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 1 -- 172.21.15.37:0/1138469426 wait complete. 2024-03-25T18:13:26.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:26.823+0000 7fd3f58f6640 1 librados: shutdown 2024-03-25T18:13:26.836 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:30.204 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:06:05.549 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.209 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:06:05.555 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.239 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:06:05.585 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.262 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:06:05.575 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.267 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:06:05.581 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.268 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:06:05.614 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.272 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:06:05.618 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.294 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:06:05.607 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:30.306 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:06:05.651 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:33.643 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:33.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.759+0000 7f3930615640 10 monclient: get_monmap_and_config 2024-03-25T18:13:33.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.759+0000 7f3930615640 10 monclient: build_initial_monmap 2024-03-25T18:13:33.761 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.759+0000 7f3930615640 10 monclient: monmap: 2024-03-25T18:13:33.761 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:33.761 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:33.761 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:13:33.760495+0000 2024-03-25T18:13:33.761 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:13:33.760495+0000 2024-03-25T18:13:33.761 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:33.761 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:33.761 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:33.761 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:13:33.761 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:13:33.761 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:33.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.760+0000 7f3930615640 1 Processor -- start 2024-03-25T18:13:33.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.760+0000 7f3930615640 1 -- start start 2024-03-25T18:13:33.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.760+0000 7f3930615640 10 monclient: init 2024-03-25T18:13:33.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.760+0000 7f3930615640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:33.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.760+0000 7f3930615640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:33.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.760+0000 7f3930615640 10 monclient(hunting): picked mon.noname-a con 0x7f39280a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:33.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.760+0000 7f3930615640 10 monclient(hunting): picked mon.noname-c con 0x7f392813f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:33.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.761+0000 7f3930615640 10 monclient(hunting): picked mon.noname-b con 0x7f39281430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:33.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.761+0000 7f3930615640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3928142520 con 0x7f39280a98f0 2024-03-25T18:13:33.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.761+0000 7f3930615640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39280b28a0 con 0x7f392813f710 2024-03-25T18:13:33.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.761+0000 7f3930615640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39280b2a30 con 0x7f39281430c0 2024-03-25T18:13:33.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.761+0000 7f3930615640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:33.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.761+0000 7f3930615640 10 monclient(hunting): authenticate will time out at 7169.013337s 2024-03-25T18:13:33.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.772+0000 7f392ebb2640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f39281430c0 0x7f392814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:58496/0 (socket says 172.21.15.37:58496) 2024-03-25T18:13:33.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.772+0000 7f392ebb2640 1 -- 172.21.15.37:0/1951353036 learned_addr learned my addr 172.21.15.37:0/1951353036 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:33.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.784+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4158265539 0 0) 0x7f39280b2a30 con 0x7f39281430c0 2024-03-25T18:13:33.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.784+0000 7f392d3af640 10 monclient(hunting): _negotiate 2024-03-25T18:13:33.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.784+0000 7f392d3af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:33.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.784+0000 7f392d3af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:33.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.784+0000 7f392d3af640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:13:33.785 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.784+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3914003520 con 0x7f39281430c0 2024-03-25T18:13:33.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.787+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3548984480 0 0) 0x7f3928142520 con 0x7f39280a98f0 2024-03-25T18:13:33.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.787+0000 7f392d3af640 10 monclient(hunting): _negotiate 2024-03-25T18:13:33.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.787+0000 7f392d3af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:33.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.787+0000 7f392d3af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:33.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.787+0000 7f392d3af640 10 monclient(hunting): my global_id is 4485 2024-03-25T18:13:33.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.787+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39280b2a30 con 0x7f39280a98f0 2024-03-25T18:13:33.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.788+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044784555 0 0) 0x7f39280b28a0 con 0x7f392813f710 2024-03-25T18:13:33.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.788+0000 7f392d3af640 10 monclient(hunting): _negotiate 2024-03-25T18:13:33.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.788+0000 7f392d3af640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:33.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.788+0000 7f392d3af640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:33.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.788+0000 7f392d3af640 10 monclient(hunting): my global_id is 4486 2024-03-25T18:13:33.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.788+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3928142520 con 0x7f392813f710 2024-03-25T18:13:33.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2228546013 0 0) 0x7f3914003520 con 0x7f39281430c0 2024-03-25T18:13:33.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 >> v1:172.21.15.117:6789/0 conn(0x7f392813f710 legacy=0x7f39281429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:33.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 >> v1:172.21.15.37:6789/0 conn(0x7f39280a98f0 legacy=0x7f39280a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:33.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 10 monclient: _finish_hunting 0 2024-03-25T18:13:33.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 1 monclient: found mon.noname-b 2024-03-25T18:13:33.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:33.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:13:33.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39280b44c0 con 0x7f39281430c0 2024-03-25T18:13:33.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 10 monclient: _finish_auth 0 2024-03-25T18:13:33.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 10 monclient: _check_auth_tickets 2024-03-25T18:13:33.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:33.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3920001a10 con 0x7f39281430c0 2024-03-25T18:13:33.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f3930615640 5 monclient: authenticate success, global_id 4481 2024-03-25T18:13:33.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f3930615640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:33.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f392d3af640 10 monclient: dump: 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:33.799 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.796+0000 7f3930615640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:33.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.806+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3920001b70 con 0x7f39281430c0 2024-03-25T18:13:33.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.806+0000 7f392d3af640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:33.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.806+0000 7f392d3af640 1 -- 172.21.15.37:0/1951353036 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3920001f80 con 0x7f39281430c0 2024-03-25T18:13:33.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.806+0000 7f392d3af640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:33.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.806+0000 7f392d3af640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:33.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.806+0000 7f392d3af640 10 monclient: dump: 2024-03-25T18:13:33.807 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:33.807 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:33.808 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:33.808 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:33.808 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:33.808 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:33.808 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:33.808 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:33.808 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:33.808 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:33.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.806+0000 7f3930615640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:33.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.806+0000 7f3930615640 4 set_mon_vals no callback set 2024-03-25T18:13:33.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.807+0000 7f3930615640 10 monclient: shutdown 2024-03-25T18:13:33.810 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.807+0000 7f3930615640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.807+0000 7f3930615640 1 -- 172.21.15.37:0/1951353036 >> v1:172.21.15.37:6790/0 conn(0x7f39281430c0 legacy=0x7f392814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.807+0000 7f3930615640 1 -- 172.21.15.37:0/1951353036 shutdown_connections 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.807+0000 7f3930615640 1 -- 172.21.15.37:0/1951353036 >> 172.21.15.37:0/1951353036 conn(0x7f39280b7ea0 msgr2=0x7f39280b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.807+0000 7f3930615640 1 -- 172.21.15.37:0/1951353036 shutdown_connections 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.807+0000 7f3930615640 1 -- 172.21.15.37:0/1951353036 wait complete. 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.808+0000 7f3930615640 10 monclient: build_initial_monmap 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.808+0000 7f3930615640 10 monclient: monmap: 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:33.811 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.808+0000 7f3930615640 1 librados: starting msgr at 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.808+0000 7f3930615640 1 librados: starting objecter 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.808+0000 7f3930615640 1 Processor -- start 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.808+0000 7f3930615640 1 -- start start 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.808+0000 7f3930615640 1 librados: setting wanted keys 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.808+0000 7f3930615640 1 librados: calling monclient init 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.808+0000 7f3930615640 10 monclient: init 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.809+0000 7f3930615640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.809+0000 7f3930615640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.809+0000 7f3930615640 10 monclient(hunting): picked mon.noname-b con 0x7f39280a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.809+0000 7f3930615640 10 monclient(hunting): picked mon.noname-c con 0x7f392813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.809+0000 7f3930615640 10 monclient(hunting): picked mon.noname-a con 0x7f39281430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.809+0000 7f3930615640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3928136ad0 con 0x7f39281430c0 2024-03-25T18:13:33.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.809+0000 7f3930615640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39280b7520 con 0x7f39280a98f0 2024-03-25T18:13:33.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.809+0000 7f3930615640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3928135810 con 0x7f392813f710 2024-03-25T18:13:33.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.809+0000 7f3930615640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:33.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.809+0000 7f3930615640 10 monclient(hunting): authenticate will time out at 7169.061808s 2024-03-25T18:13:33.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.814+0000 7f392dbb0640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f392813f710 0x7f39280b5a70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:58540/0 (socket says 172.21.15.37:58540) 2024-03-25T18:13:33.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.814+0000 7f392dbb0640 1 -- 172.21.15.37:0/7777469 learned_addr learned my addr 172.21.15.37:0/7777469 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:33.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.818+0000 7f391effd640 10 client.?.objecter ms_handle_connect 0x7f39281430c0 2024-03-25T18:13:33.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.818+0000 7f391effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:33.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.821+0000 7f391effd640 10 client.?.objecter ms_handle_connect 0x7f392813f710 2024-03-25T18:13:33.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.821+0000 7f391effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:33.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.827+0000 7f391effd640 10 client.?.objecter ms_handle_connect 0x7f39280a98f0 2024-03-25T18:13:33.828 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.827+0000 7f391effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:33.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.827+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 91729663 0 0) 0x7f3928136ad0 con 0x7f39281430c0 2024-03-25T18:13:33.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.827+0000 7f391effd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:33.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.827+0000 7f391effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:33.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.827+0000 7f391effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:33.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.827+0000 7f391effd640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:13:33.829 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.828+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f390c004eb0 con 0x7f39281430c0 2024-03-25T18:13:33.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.830+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1589070159 0 0) 0x7f3928135810 con 0x7f392813f710 2024-03-25T18:13:33.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.830+0000 7f391effd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:33.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.830+0000 7f391effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:33.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.830+0000 7f391effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:33.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.830+0000 7f391effd640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:13:33.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.830+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3928136ad0 con 0x7f392813f710 2024-03-25T18:13:33.838 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.837+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738770407 0 0) 0x7f39280b7520 con 0x7f39280a98f0 2024-03-25T18:13:33.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.837+0000 7f391effd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:33.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.837+0000 7f391effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:33.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.837+0000 7f391effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:33.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.837+0000 7f391effd640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:13:33.839 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.837+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3928135810 con 0x7f39280a98f0 2024-03-25T18:13:33.840 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.839+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3464328928 0 0) 0x7f3928136ad0 con 0x7f392813f710 2024-03-25T18:13:33.840 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.839+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f39280b7520 con 0x7f392813f710 2024-03-25T18:13:33.840 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.839+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f39180019f0 con 0x7f392813f710 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.839+0000 7f391effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.839+0000 7f391effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.839+0000 7f391effd640 10 monclient(hunting): dump: 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:33.841 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.839+0000 7f391effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:33.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1658797023 0 0) 0x7f3928135810 con 0x7f39280a98f0 2024-03-25T18:13:33.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3928136ad0 con 0x7f39280a98f0 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3924001930 con 0x7f39280a98f0 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 10 monclient(hunting): dump: 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:33.851 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2631624401 0 0) 0x7f390c004eb0 con 0x7f39281430c0 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3928135810 con 0x7f39281430c0 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3920001e80 con 0x7f39281430c0 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.849+0000 7f391effd640 10 monclient(hunting): dump: 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:33.852 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 153228505 0 0) 0x7f39280b7520 con 0x7f392813f710 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 >> v1:172.21.15.117:6789/0 conn(0x7f39280a98f0 legacy=0x7f3928135100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 >> v1:172.21.15.37:6789/0 conn(0x7f39281430c0 legacy=0x7f3928130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f391effd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f391effd640 1 monclient: found mon.c 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f391effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f391effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39280b6180 con 0x7f392813f710 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f391effd640 10 monclient: _finish_auth 0 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f391effd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f391effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f3930615640 5 monclient: authenticate success, global_id 4484 2024-03-25T18:13:33.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f3930615640 10 monclient: _renew_subs 2024-03-25T18:13:33.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f3930615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:33.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f3930615640 1 -- 172.21.15.37:0/7777469 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39280b7a40 con 0x7f392813f710 2024-03-25T18:13:33.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f3930615640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:33.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f3930615640 10 monclient: _renew_subs 2024-03-25T18:13:33.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f3930615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:33.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f3930615640 1 -- 172.21.15.37:0/7777469 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3928131840 con 0x7f392813f710 2024-03-25T18:13:33.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.851+0000 7f3930615640 1 librados: init done 2024-03-25T18:13:33.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.852+0000 7f3930615640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:33.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.852+0000 7f3930615640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:33.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.852+0000 7f3930615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:33.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:33.852+0000 7f3930615640 1 -- 172.21.15.37:0/7777469 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3928136f60 con 0x7f392813f710 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.065+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3918003610 con 0x7f392813f710 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.065+0000 7f391effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.065+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3918003800 con 0x7f392813f710 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.065+0000 7f391effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.065+0000 7f391effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.065+0000 7f391effd640 10 monclient: dump: 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:34.066 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:34.067 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:34.067 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:34.067 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:34.067 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:34.067 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.065+0000 7f392cbae640 4 set_mon_vals no callback set 2024-03-25T18:13:34.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.169+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f3918003960 con 0x7f392813f710 2024-03-25T18:13:34.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.170+0000 7f392e3b1640 10 monclient: get_auth_request con 0x7f390c06e0f0 auth_method 0 2024-03-25T18:13:34.171 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.170+0000 7f391effd640 10 client.4484.objecter ms_handle_connect 0x7f390c06e0f0 2024-03-25T18:13:34.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.178+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f3918054650 con 0x7f392813f710 2024-03-25T18:13:34.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.178+0000 7f391effd640 10 client.4484.objecter ms_dispatch 0x7f3928000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:34.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.178+0000 7f391effd640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:34.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.178+0000 7f391effd640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:34.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.178+0000 7f391effd640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-25T18:13:34.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.178+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f391804ff90 con 0x7f392813f710 2024-03-25T18:13:34.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.178+0000 7f391effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:34.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.178+0000 7f391effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:34.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.423+0000 7f3930615640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:34.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.423+0000 7f3930615640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:34.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.423+0000 7f3930615640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:34.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.423+0000 7f3930615640 1 -- 172.21.15.37:0/7777469 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f39280b6310 con 0x7f392813f710 2024-03-25T18:13:34.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.443+0000 7f391effd640 1 -- 172.21.15.37:0/7777469 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f3918052090 con 0x7f392813f710 2024-03-25T18:13:34.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.443+0000 7f391effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:34.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.443+0000 7f391effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:34.444 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:13:34.445 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:34.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 10 librados: watch_flush enter 2024-03-25T18:13:34.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 10 librados: watch_flush exit 2024-03-25T18:13:34.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-25T18:13:34.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 10 client.4484.objecter successfully canceled tick 2024-03-25T18:13:34.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 1 -- 172.21.15.37:0/7777469 >> v1:172.21.15.117:6800/3703993921 conn(0x7f390c06e0f0 legacy=0x7f390c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 10 monclient: shutdown 2024-03-25T18:13:34.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:34.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 1 -- 172.21.15.37:0/7777469 >> v1:172.21.15.37:6790/0 conn(0x7f392813f710 legacy=0x7f39280b5a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:34.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 1 -- 172.21.15.37:0/7777469 shutdown_connections 2024-03-25T18:13:34.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 1 -- 172.21.15.37:0/7777469 >> 172.21.15.37:0/7777469 conn(0x7f39280b7ea0 msgr2=0x7f3928074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:34.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 1 -- 172.21.15.37:0/7777469 shutdown_connections 2024-03-25T18:13:34.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 1 -- 172.21.15.37:0/7777469 wait complete. 2024-03-25T18:13:34.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:34.445+0000 7f3930615640 1 librados: shutdown 2024-03-25T18:13:34.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:41.267 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:41.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.385+0000 7f8ecab8d640 10 monclient: get_monmap_and_config 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.385+0000 7f8ecab8d640 10 monclient: build_initial_monmap 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.385+0000 7f8ecab8d640 10 monclient: monmap: 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:13:41.386542+0000 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:13:41.386542+0000 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:13:41.387 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:41.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.386+0000 7f8ecab8d640 1 Processor -- start 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.386+0000 7f8ecab8d640 1 -- start start 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.386+0000 7f8ecab8d640 10 monclient: init 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.387+0000 7f8ecab8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.387+0000 7f8ecab8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.387+0000 7f8ecab8d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec40a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.387+0000 7f8ecab8d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ec413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.387+0000 7f8ecab8d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ec41430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.387+0000 7f8ecab8d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec4142520 con 0x7f8ec413f710 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.387+0000 7f8ecab8d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec40b28a0 con 0x7f8ec40a98f0 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.387+0000 7f8ecab8d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec40b2a30 con 0x7f8ec41430c0 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.387+0000 7f8ecab8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:41.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.387+0000 7f8ecab8d640 10 monclient(hunting): authenticate will time out at 7176.639904s 2024-03-25T18:13:41.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.401+0000 7f8ec8929640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f8ec40a98f0 0x7f8ec40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:54002/0 (socket says 172.21.15.37:54002) 2024-03-25T18:13:41.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.401+0000 7f8ec8929640 1 -- 172.21.15.37:0/2513845909 learned_addr learned my addr 172.21.15.37:0/2513845909 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:41.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.408+0000 7f8ec37fe640 1 -- 172.21.15.37:0/2513845909 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3608215777 0 0) 0x7f8ec40b28a0 con 0x7f8ec40a98f0 2024-03-25T18:13:41.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.408+0000 7f8ec37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:41.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.408+0000 7f8ec37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:41.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.408+0000 7f8ec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:41.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.408+0000 7f8ec37fe640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:13:41.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.408+0000 7f8ec37fe640 1 -- 172.21.15.37:0/2513845909 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8eb0003520 con 0x7f8ec40a98f0 2024-03-25T18:13:41.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.421+0000 7f8ec37fe640 1 -- 172.21.15.37:0/2513845909 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 474705264 0 0) 0x7f8ec40b2a30 con 0x7f8ec41430c0 2024-03-25T18:13:41.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.421+0000 7f8ec37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:13:41.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.421+0000 7f8ec37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:41.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.421+0000 7f8ec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:41.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.421+0000 7f8ec37fe640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:13:41.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.421+0000 7f8ec37fe640 1 -- 172.21.15.37:0/2513845909 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec40b28a0 con 0x7f8ec41430c0 2024-03-25T18:13:41.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 1 -- 172.21.15.37:0/2513845909 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1890420117 0 0) 0x7f8eb0003520 con 0x7f8ec40a98f0 2024-03-25T18:13:41.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 1 -- 172.21.15.37:0/2513845909 >> v1:172.21.15.37:6790/0 conn(0x7f8ec41430c0 legacy=0x7f8ec414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:41.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 1 -- 172.21.15.37:0/2513845909 >> v1:172.21.15.37:6789/0 conn(0x7f8ec413f710 legacy=0x7f8ec41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:41.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:13:41.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 1 monclient: found mon.noname-c 2024-03-25T18:13:41.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:41.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:13:41.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 1 -- 172.21.15.37:0/2513845909 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec40b44c0 con 0x7f8ec40a98f0 2024-03-25T18:13:41.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 10 monclient: _finish_auth 0 2024-03-25T18:13:41.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 1 -- 172.21.15.37:0/2513845909 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8eb4001690 con 0x7f8ec40a98f0 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ec37fe640 10 monclient: dump: 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:41.430 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:41.431 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:41.431 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:41.431 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:41.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ecab8d640 5 monclient: authenticate success, global_id 4492 2024-03-25T18:13:41.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.427+0000 7f8ecab8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.441+0000 7f8ec37fe640 1 -- 172.21.15.37:0/2513845909 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8eb40017f0 con 0x7f8ec40a98f0 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.441+0000 7f8ec37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.441+0000 7f8ec37fe640 1 -- 172.21.15.37:0/2513845909 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8eb4001c00 con 0x7f8ec40a98f0 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.441+0000 7f8ecab8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.441+0000 7f8ecab8d640 4 set_mon_vals no callback set 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.441+0000 7f8ec37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.441+0000 7f8ec37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.441+0000 7f8ec37fe640 10 monclient: dump: 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:41.442 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:41.443 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:41.443 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:41.443 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:41.443 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:41.443 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:41.443 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:41.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.442+0000 7f8ecab8d640 10 monclient: shutdown 2024-03-25T18:13:41.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.442+0000 7f8ecab8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:41.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.442+0000 7f8ecab8d640 1 -- 172.21.15.37:0/2513845909 >> v1:172.21.15.117:6789/0 conn(0x7f8ec40a98f0 legacy=0x7f8ec40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:41.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.442+0000 7f8ecab8d640 1 -- 172.21.15.37:0/2513845909 shutdown_connections 2024-03-25T18:13:41.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.442+0000 7f8ecab8d640 1 -- 172.21.15.37:0/2513845909 >> 172.21.15.37:0/2513845909 conn(0x7f8ec40b7ea0 msgr2=0x7f8ec40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:41.444 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.442+0000 7f8ecab8d640 1 -- 172.21.15.37:0/2513845909 shutdown_connections 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.442+0000 7f8ecab8d640 1 -- 172.21.15.37:0/2513845909 wait complete. 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.443+0000 7f8ecab8d640 10 monclient: build_initial_monmap 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.443+0000 7f8ecab8d640 10 monclient: monmap: 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.443+0000 7f8ecab8d640 1 librados: starting msgr at 2024-03-25T18:13:41.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.443+0000 7f8ecab8d640 1 librados: starting objecter 2024-03-25T18:13:41.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.443+0000 7f8ecab8d640 1 Processor -- start 2024-03-25T18:13:41.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.443+0000 7f8ecab8d640 1 -- start start 2024-03-25T18:13:41.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.443+0000 7f8ecab8d640 1 librados: setting wanted keys 2024-03-25T18:13:41.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.443+0000 7f8ecab8d640 1 librados: calling monclient init 2024-03-25T18:13:41.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.443+0000 7f8ecab8d640 10 monclient: init 2024-03-25T18:13:41.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.444+0000 7f8ecab8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:41.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.444+0000 7f8ecab8d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:41.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.444+0000 7f8ecab8d640 10 monclient(hunting): picked mon.noname-b con 0x7f8ec40a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:41.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.444+0000 7f8ecab8d640 10 monclient(hunting): picked mon.noname-a con 0x7f8ec413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:41.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.444+0000 7f8ecab8d640 10 monclient(hunting): picked mon.noname-c con 0x7f8ec41430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:41.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.444+0000 7f8ecab8d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec4138a50 con 0x7f8ec413f710 2024-03-25T18:13:41.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.444+0000 7f8ecab8d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec40ab9e0 con 0x7f8ec40a98f0 2024-03-25T18:13:41.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.444+0000 7f8ecab8d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8ec41d7880 con 0x7f8ec41430c0 2024-03-25T18:13:41.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.444+0000 7f8ecab8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:41.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.444+0000 7f8ecab8d640 10 monclient(hunting): authenticate will time out at 7176.696563s 2024-03-25T18:13:41.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.446+0000 7f8ec3fff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f8ec413f710 0x7f8ec41d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55740/0 (socket says 172.21.15.37:55740) 2024-03-25T18:13:41.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.446+0000 7f8ec3fff640 1 -- 172.21.15.37:0/1868583874 learned_addr learned my addr 172.21.15.37:0/1868583874 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:41.449 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.449+0000 7f8ec17fa640 10 client.?.objecter ms_handle_connect 0x7f8ec413f710 2024-03-25T18:13:41.449 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.449+0000 7f8ec17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:41.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.452+0000 7f8ec17fa640 10 client.?.objecter ms_handle_connect 0x7f8ec41430c0 2024-03-25T18:13:41.453 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.452+0000 7f8ec17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:41.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.460+0000 7f8ec17fa640 10 client.?.objecter ms_handle_connect 0x7f8ec40a98f0 2024-03-25T18:13:41.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.460+0000 7f8ec17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:41.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.461+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 737076221 0 0) 0x7f8ec4138a50 con 0x7f8ec413f710 2024-03-25T18:13:41.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.461+0000 7f8ec17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:41.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.461+0000 7f8ec17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:41.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.461+0000 7f8ec17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:41.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.461+0000 7f8ec17fa640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:13:41.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.461+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ea4004eb0 con 0x7f8ec413f710 2024-03-25T18:13:41.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.466+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 968256021 0 0) 0x7f8ec41d7880 con 0x7f8ec41430c0 2024-03-25T18:13:41.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.466+0000 7f8ec17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:41.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.466+0000 7f8ec17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:41.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.466+0000 7f8ec17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:41.467 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.466+0000 7f8ec17fa640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:13:41.468 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.466+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec4138a50 con 0x7f8ec41430c0 2024-03-25T18:13:41.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.473+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116223216 0 0) 0x7f8ec40ab9e0 con 0x7f8ec40a98f0 2024-03-25T18:13:41.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.473+0000 7f8ec17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:13:41.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.473+0000 7f8ec17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:41.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.473+0000 7f8ec17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:41.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.473+0000 7f8ec17fa640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:13:41.474 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.473+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8ec41d7880 con 0x7f8ec40a98f0 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.476+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1738845640 0 0) 0x7f8ec4138a50 con 0x7f8ec41430c0 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.476+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8ec40ab9e0 con 0x7f8ec41430c0 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.476+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8ebc001b90 con 0x7f8ec41430c0 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.476+0000 7f8ec17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.476+0000 7f8ec17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.476+0000 7f8ec17fa640 10 monclient(hunting): dump: 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:41.478 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:41.479 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:41.479 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:41.479 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:41.479 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.476+0000 7f8ec17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:41.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.491+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1160846793 0 0) 0x7f8ec40ab9e0 con 0x7f8ec41430c0 2024-03-25T18:13:41.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 >> v1:172.21.15.117:6789/0 conn(0x7f8ec40a98f0 legacy=0x7f8ec4138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:41.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 >> v1:172.21.15.37:6789/0 conn(0x7f8ec413f710 legacy=0x7f8ec41d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:41.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ec17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:13:41.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ec17fa640 1 monclient: found mon.c 2024-03-25T18:13:41.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ec17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:41.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ec17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:41.494 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec41d84b0 con 0x7f8ec41430c0 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ec17fa640 10 monclient: _finish_auth 0 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ec17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ec17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ecab8d640 5 monclient: authenticate success, global_id 4490 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ecab8d640 10 monclient: _renew_subs 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ecab8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ecab8d640 1 -- 172.21.15.37:0/1868583874 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec41d51f0 con 0x7f8ec41430c0 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ecab8d640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ecab8d640 10 monclient: _renew_subs 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ecab8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ecab8d640 1 -- 172.21.15.37:0/1868583874 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ec4138e60 con 0x7f8ec41430c0 2024-03-25T18:13:41.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.492+0000 7f8ecab8d640 1 librados: init done 2024-03-25T18:13:41.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.493+0000 7f8ecab8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:41.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.493+0000 7f8ecab8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:41.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.493+0000 7f8ecab8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:41.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.493+0000 7f8ecab8d640 1 -- 172.21.15.37:0/1868583874 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ec41d76e0 con 0x7f8ec41430c0 2024-03-25T18:13:41.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.821+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8ebc002040 con 0x7f8ec41430c0 2024-03-25T18:13:41.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.821+0000 7f8ec17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:41.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.821+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8ebc003990 con 0x7f8ec41430c0 2024-03-25T18:13:41.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.821+0000 7f8ec17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:41.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.821+0000 7f8ec17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:41.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.821+0000 7f8ec17fa640 10 monclient: dump: 2024-03-25T18:13:41.823 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:41.823 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:41.823 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:41.823 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:41.823 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:41.824 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:41.824 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:41.824 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:41.824 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:41.824 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:41.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.821+0000 7f8ec2ffd640 4 set_mon_vals no callback set 2024-03-25T18:13:41.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.822+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f8ebc0199a0 con 0x7f8ec41430c0 2024-03-25T18:13:41.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.823+0000 7f8ec8929640 10 monclient: get_auth_request con 0x7f8ea406de20 auth_method 0 2024-03-25T18:13:41.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.823+0000 7f8ec17fa640 10 client.4490.objecter ms_handle_connect 0x7f8ea406de20 2024-03-25T18:13:41.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.828+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f8ebc053670 con 0x7f8ec41430c0 2024-03-25T18:13:41.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.828+0000 7f8ec17fa640 10 client.4490.objecter ms_dispatch 0x7f8ec4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:41.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.828+0000 7f8ec17fa640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:41.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.828+0000 7f8ec17fa640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:41.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.829+0000 7f8ec17fa640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-25T18:13:41.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.829+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8ebc0b08b0 con 0x7f8ec41430c0 2024-03-25T18:13:41.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.829+0000 7f8ec17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:41.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:41.829+0000 7f8ec17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:42.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.085+0000 7f8ecab8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:42.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.085+0000 7f8ecab8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:42.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.085+0000 7f8ecab8d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:13:42.086 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.085+0000 7f8ecab8d640 1 -- 172.21.15.37:0/1868583874 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8ec40627b0 con 0x7f8ec41430c0 2024-03-25T18:13:42.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.188+0000 7f8ec17fa640 1 -- 172.21.15.37:0/1868583874 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f8ebc04efb0 con 0x7f8ec41430c0 2024-03-25T18:13:42.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.188+0000 7f8ec17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:42.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.188+0000 7f8ec17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:42.190 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:13:42.190 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:42.192 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 10 librados: watch_flush enter 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 10 librados: watch_flush exit 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 10 client.4490.objecter successfully canceled tick 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 1 -- 172.21.15.37:0/1868583874 >> v1:172.21.15.117:6800/3703993921 conn(0x7f8ea406de20 legacy=0x7f8ea40702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 10 monclient: shutdown 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 1 -- 172.21.15.37:0/1868583874 >> v1:172.21.15.37:6790/0 conn(0x7f8ec41430c0 legacy=0x7f8ec41e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 1 -- 172.21.15.37:0/1868583874 shutdown_connections 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 1 -- 172.21.15.37:0/1868583874 >> 172.21.15.37:0/1868583874 conn(0x7f8ec40b7ea0 msgr2=0x7f8ec4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 1 -- 172.21.15.37:0/1868583874 shutdown_connections 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 1 -- 172.21.15.37:0/1868583874 wait complete. 2024-03-25T18:13:42.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:42.191+0000 7f8ecab8d640 1 librados: shutdown 2024-03-25T18:13:42.205 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:49.011 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:49.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.128+0000 7feb8e85c640 10 monclient: get_monmap_and_config 2024-03-25T18:13:49.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.128+0000 7feb8e85c640 10 monclient: build_initial_monmap 2024-03-25T18:13:49.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.128+0000 7feb8e85c640 10 monclient: monmap: 2024-03-25T18:13:49.129 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:49.129 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:49.129 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:13:49.129066+0000 2024-03-25T18:13:49.129 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:13:49.129066+0000 2024-03-25T18:13:49.130 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:49.130 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:49.130 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:49.130 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:13:49.130 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:13:49.130 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:49.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.129+0000 7feb8e85c640 1 Processor -- start 2024-03-25T18:13:49.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.129+0000 7feb8e85c640 1 -- start start 2024-03-25T18:13:49.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.129+0000 7feb8e85c640 10 monclient: init 2024-03-25T18:13:49.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.130+0000 7feb8e85c640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:49.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.130+0000 7feb8e85c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:13:49.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.130+0000 7feb8e85c640 10 monclient(hunting): picked mon.noname-a con 0x7feb880a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:49.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.130+0000 7feb8e85c640 10 monclient(hunting): picked mon.noname-b con 0x7feb8813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:49.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.130+0000 7feb8e85c640 10 monclient(hunting): picked mon.noname-c con 0x7feb881430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:49.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.130+0000 7feb8e85c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb88142520 con 0x7feb880a98f0 2024-03-25T18:13:49.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.130+0000 7feb8e85c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb880b28a0 con 0x7feb881430c0 2024-03-25T18:13:49.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.130+0000 7feb8e85c640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb880b2a30 con 0x7feb8813f710 2024-03-25T18:13:49.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.130+0000 7feb8e85c640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:49.132 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.130+0000 7feb8e85c640 10 monclient(hunting): authenticate will time out at 7184.382690s 2024-03-25T18:13:49.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.140+0000 7feb8cdf9640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7feb881430c0 0x7feb8814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:58130/0 (socket says 172.21.15.37:58130) 2024-03-25T18:13:49.142 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.140+0000 7feb8cdf9640 1 -- 172.21.15.37:0/2422577169 learned_addr learned my addr 172.21.15.37:0/2422577169 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:49.147 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.146+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435732371 0 0) 0x7feb880b28a0 con 0x7feb881430c0 2024-03-25T18:13:49.147 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.146+0000 7feb86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:49.147 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.146+0000 7feb86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:49.147 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.146+0000 7feb86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:49.147 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.146+0000 7feb86ffd640 10 monclient(hunting): my global_id is 4498 2024-03-25T18:13:49.147 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.146+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb70003520 con 0x7feb881430c0 2024-03-25T18:13:49.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.155+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2633512028 0 0) 0x7feb88142520 con 0x7feb880a98f0 2024-03-25T18:13:49.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.155+0000 7feb86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:49.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.155+0000 7feb86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:49.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.155+0000 7feb86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:49.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.155+0000 7feb86ffd640 10 monclient(hunting): my global_id is 4497 2024-03-25T18:13:49.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.155+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb880b28a0 con 0x7feb880a98f0 2024-03-25T18:13:49.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.166+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 543517175 0 0) 0x7feb880b2a30 con 0x7feb8813f710 2024-03-25T18:13:49.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.166+0000 7feb86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:49.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.166+0000 7feb86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:49.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.166+0000 7feb86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:49.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.166+0000 7feb86ffd640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:13:49.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.166+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb88142520 con 0x7feb8813f710 2024-03-25T18:13:49.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1250339204 0 0) 0x7feb70003520 con 0x7feb881430c0 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 >> v1:172.21.15.37:6790/0 conn(0x7feb8813f710 legacy=0x7feb881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 >> v1:172.21.15.37:6789/0 conn(0x7feb880a98f0 legacy=0x7feb880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 1 monclient: found mon.noname-c 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb880b44c0 con 0x7feb881430c0 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7feb80001a10 con 0x7feb881430c0 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:13:49.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb86ffd640 10 monclient: dump: 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb8e85c640 5 monclient: authenticate success, global_id 4498 2024-03-25T18:13:49.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.167+0000 7feb8e85c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.175+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb80001b70 con 0x7feb881430c0 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.175+0000 7feb86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.175+0000 7feb86ffd640 1 -- 172.21.15.37:0/2422577169 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7feb80001f80 con 0x7feb881430c0 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.175+0000 7feb86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.175+0000 7feb86ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.175+0000 7feb86ffd640 10 monclient: dump: 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:49.176 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:49.177 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:49.177 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:49.177 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:49.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.175+0000 7feb8e85c640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:49.177 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.175+0000 7feb8e85c640 4 set_mon_vals no callback set 2024-03-25T18:13:49.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.176+0000 7feb8e85c640 10 monclient: shutdown 2024-03-25T18:13:49.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.176+0000 7feb8e85c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:49.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.176+0000 7feb8e85c640 1 -- 172.21.15.37:0/2422577169 >> v1:172.21.15.117:6789/0 conn(0x7feb881430c0 legacy=0x7feb8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:49.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.176+0000 7feb8e85c640 1 -- 172.21.15.37:0/2422577169 shutdown_connections 2024-03-25T18:13:49.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.176+0000 7feb8e85c640 1 -- 172.21.15.37:0/2422577169 >> 172.21.15.37:0/2422577169 conn(0x7feb880b7ea0 msgr2=0x7feb880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:49.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.176+0000 7feb8e85c640 1 -- 172.21.15.37:0/2422577169 shutdown_connections 2024-03-25T18:13:49.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.176+0000 7feb8e85c640 1 -- 172.21.15.37:0/2422577169 wait complete. 2024-03-25T18:13:49.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.176+0000 7feb8e85c640 10 monclient: build_initial_monmap 2024-03-25T18:13:49.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.177+0000 7feb8e85c640 10 monclient: monmap: 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.177+0000 7feb8e85c640 1 librados: starting msgr at 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.177+0000 7feb8e85c640 1 librados: starting objecter 2024-03-25T18:13:49.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.177+0000 7feb8e85c640 1 Processor -- start 2024-03-25T18:13:49.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.177+0000 7feb8e85c640 1 -- start start 2024-03-25T18:13:49.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.177+0000 7feb8e85c640 1 librados: setting wanted keys 2024-03-25T18:13:49.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.177+0000 7feb8e85c640 1 librados: calling monclient init 2024-03-25T18:13:49.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.177+0000 7feb8e85c640 10 monclient: init 2024-03-25T18:13:49.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.178+0000 7feb8e85c640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:49.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.178+0000 7feb8e85c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:13:49.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.178+0000 7feb8e85c640 10 monclient(hunting): picked mon.noname-c con 0x7feb880a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:49.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.178+0000 7feb8e85c640 10 monclient(hunting): picked mon.noname-b con 0x7feb8813f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:49.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.178+0000 7feb8e85c640 10 monclient(hunting): picked mon.noname-a con 0x7feb881430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:49.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.178+0000 7feb8e85c640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb881369d0 con 0x7feb881430c0 2024-03-25T18:13:49.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.178+0000 7feb8e85c640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb88133a60 con 0x7feb8813f710 2024-03-25T18:13:49.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.178+0000 7feb8e85c640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb880ab9e0 con 0x7feb880a98f0 2024-03-25T18:13:49.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.178+0000 7feb8e85c640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:49.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.178+0000 7feb8e85c640 10 monclient(hunting): authenticate will time out at 7184.430820s 2024-03-25T18:13:49.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.192+0000 7feb87fff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7feb880a98f0 0x7feb88132c40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:39690/0 (socket says 172.21.15.37:39690) 2024-03-25T18:13:49.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.192+0000 7feb87fff640 1 -- 172.21.15.37:0/1602241748 learned_addr learned my addr 172.21.15.37:0/1602241748 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:49.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.194+0000 7feb84ff9640 10 client.?.objecter ms_handle_connect 0x7feb8813f710 2024-03-25T18:13:49.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.194+0000 7feb84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:49.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.194+0000 7feb84ff9640 10 client.?.objecter ms_handle_connect 0x7feb880a98f0 2024-03-25T18:13:49.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.194+0000 7feb84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:49.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.198+0000 7feb84ff9640 10 client.?.objecter ms_handle_connect 0x7feb881430c0 2024-03-25T18:13:49.199 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.198+0000 7feb84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:49.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.206+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930112036 0 0) 0x7feb88133a60 con 0x7feb8813f710 2024-03-25T18:13:49.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.206+0000 7feb84ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:49.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.206+0000 7feb84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:49.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.206+0000 7feb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:49.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.206+0000 7feb84ff9640 10 monclient(hunting): my global_id is 4501 2024-03-25T18:13:49.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.206+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb74004eb0 con 0x7feb8813f710 2024-03-25T18:13:49.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.207+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1753521592 0 0) 0x7feb880ab9e0 con 0x7feb880a98f0 2024-03-25T18:13:49.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.207+0000 7feb84ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:49.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.207+0000 7feb84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:49.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.207+0000 7feb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:49.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.207+0000 7feb84ff9640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:13:49.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.207+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb88133a60 con 0x7feb880a98f0 2024-03-25T18:13:49.228 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.227+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2950698827 0 0) 0x7feb881369d0 con 0x7feb881430c0 2024-03-25T18:13:49.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.227+0000 7feb84ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:13:49.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.227+0000 7feb84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:49.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.227+0000 7feb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:49.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.227+0000 7feb84ff9640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:13:49.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.227+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb880ab9e0 con 0x7feb881430c0 2024-03-25T18:13:49.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.229+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1217534689 0 0) 0x7feb74004eb0 con 0x7feb8813f710 2024-03-25T18:13:49.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.229+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb881369d0 con 0x7feb8813f710 2024-03-25T18:13:49.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.229+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7feb7c0018f0 con 0x7feb8813f710 2024-03-25T18:13:49.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.229+0000 7feb84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:49.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.229+0000 7feb84ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:13:49.230 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.229+0000 7feb84ff9640 10 monclient(hunting): dump: 2024-03-25T18:13:49.230 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:49.231 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:49.231 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.231 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.231 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:49.231 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:49.231 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:49.231 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:49.231 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:49.231 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:49.231 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.229+0000 7feb84ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:49.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2897206898 0 0) 0x7feb880ab9e0 con 0x7feb881430c0 2024-03-25T18:13:49.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb74004eb0 con 0x7feb881430c0 2024-03-25T18:13:49.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7feb80001e80 con 0x7feb881430c0 2024-03-25T18:13:49.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:49.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:49.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 10 monclient(hunting): dump: 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1110976035 0 0) 0x7feb88133a60 con 0x7feb880a98f0 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb880ab9e0 con 0x7feb880a98f0 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7feb780019b0 con 0x7feb880a98f0 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:49.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:49.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.242+0000 7feb84ff9640 10 monclient(hunting): dump: 2024-03-25T18:13:49.245 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:49.245 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:49.245 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.245 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.245 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:49.245 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:49.245 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:49.245 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:49.245 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:49.245 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:49.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 165126984 0 0) 0x7feb881369d0 con 0x7feb8813f710 2024-03-25T18:13:49.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 >> v1:172.21.15.37:6790/0 conn(0x7feb880a98f0 legacy=0x7feb88132c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 >> v1:172.21.15.37:6789/0 conn(0x7feb881430c0 legacy=0x7feb88133350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb84ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb84ff9640 1 monclient: found mon.b 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb84ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb84ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb880abcc0 con 0x7feb8813f710 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb84ff9640 10 monclient: _finish_auth 0 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb84ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb8e85c640 5 monclient: authenticate success, global_id 4501 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb8e85c640 10 monclient: _renew_subs 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb8e85c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb8e85c640 1 -- 172.21.15.37:0/1602241748 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb880abfa0 con 0x7feb8813f710 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb8e85c640 10 client.4501.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:49.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb8e85c640 10 monclient: _renew_subs 2024-03-25T18:13:49.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb8e85c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:49.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb8e85c640 1 -- 172.21.15.37:0/1602241748 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb88137e40 con 0x7feb8813f710 2024-03-25T18:13:49.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.247+0000 7feb8e85c640 1 librados: init done 2024-03-25T18:13:49.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.248+0000 7feb8e85c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:49.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.248+0000 7feb8e85c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:49.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.248+0000 7feb8e85c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:49.250 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.248+0000 7feb8e85c640 1 -- 172.21.15.37:0/1602241748 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb880627b0 con 0x7feb8813f710 2024-03-25T18:13:49.540 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.539+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb7c003610 con 0x7feb8813f710 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.539+0000 7feb84ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.539+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7feb7c003920 con 0x7feb8813f710 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.539+0000 7feb84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.539+0000 7feb84ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.539+0000 7feb84ff9640 10 monclient: dump: 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:49.541 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:49.542 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:49.542 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:49.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.539+0000 7feb867fc640 4 set_mon_vals no callback set 2024-03-25T18:13:49.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.540+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7feb7c019930 con 0x7feb8813f710 2024-03-25T18:13:49.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.541+0000 7feb87fff640 10 monclient: get_auth_request con 0x7feb7406e0c0 auth_method 0 2024-03-25T18:13:49.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.541+0000 7feb84ff9640 10 client.4501.objecter ms_handle_connect 0x7feb7406e0c0 2024-03-25T18:13:49.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.544+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7feb7c0535d0 con 0x7feb8813f710 2024-03-25T18:13:49.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.544+0000 7feb84ff9640 10 client.4501.objecter ms_dispatch 0x7feb88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:49.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.544+0000 7feb84ff9640 3 client.4501.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:49.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.545+0000 7feb84ff9640 3 client.4501.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:49.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.545+0000 7feb84ff9640 20 client.4501.objecter dump_active .. 0 homeless 2024-03-25T18:13:49.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.545+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb7c04ef10 con 0x7feb8813f710 2024-03-25T18:13:49.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.545+0000 7feb84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:49.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.545+0000 7feb84ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:49.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.773+0000 7feb8e85c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:49.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.773+0000 7feb8e85c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:49.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.773+0000 7feb8e85c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:49.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.773+0000 7feb8e85c640 1 -- 172.21.15.37:0/1602241748 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb8813dfd0 con 0x7feb8813f710 2024-03-25T18:13:49.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.811+0000 7feb84ff9640 1 -- 172.21.15.37:0/1602241748 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7feb7c051010 con 0x7feb8813f710 2024-03-25T18:13:49.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.811+0000 7feb84ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:49.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.811+0000 7feb84ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:49.813 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:13:49.813 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:49.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 10 librados: watch_flush enter 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 10 librados: watch_flush exit 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 20 client.4501.objecter shutdown clearing up homeless session... 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 10 client.4501.objecter successfully canceled tick 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 1 -- 172.21.15.37:0/1602241748 >> v1:172.21.15.117:6800/3703993921 conn(0x7feb7406e0c0 legacy=0x7feb74070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 10 monclient: shutdown 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 1 -- 172.21.15.37:0/1602241748 >> v1:172.21.15.117:6789/0 conn(0x7feb8813f710 legacy=0x7feb881362c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 1 -- 172.21.15.37:0/1602241748 shutdown_connections 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 1 -- 172.21.15.37:0/1602241748 >> 172.21.15.37:0/1602241748 conn(0x7feb880b7ea0 msgr2=0x7feb88074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 1 -- 172.21.15.37:0/1602241748 shutdown_connections 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 1 -- 172.21.15.37:0/1602241748 wait complete. 2024-03-25T18:13:49.816 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:49.814+0000 7feb8e85c640 1 librados: shutdown 2024-03-25T18:13:49.827 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:13:50.251 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:06:25.596 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.257 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:06:25.603 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.287 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:06:25.632 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.311 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:06:25.624 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.316 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:06:25.629 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.340 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:06:25.653 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.344 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:06:25.690 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.353 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:06:25.699 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:50.382 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:06:25.728 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:13:56.633 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:13:56.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.751+0000 7febd09b5640 10 monclient: get_monmap_and_config 2024-03-25T18:13:56.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.751+0000 7febd09b5640 10 monclient: build_initial_monmap 2024-03-25T18:13:56.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.751+0000 7febd09b5640 10 monclient: monmap: 2024-03-25T18:13:56.752 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:56.752 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:56.752 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:13:56.752049+0000 2024-03-25T18:13:56.753 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:13:56.752049+0000 2024-03-25T18:13:56.753 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:56.753 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:56.753 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:56.753 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:13:56.753 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:13:56.753 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:56.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 1 Processor -- start 2024-03-25T18:13:56.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 1 -- start start 2024-03-25T18:13:56.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 10 monclient: init 2024-03-25T18:13:56.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:56.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:13:56.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 10 monclient(hunting): picked mon.noname-c con 0x7febcc0a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:56.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 10 monclient(hunting): picked mon.noname-b con 0x7febcc13f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:56.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 10 monclient(hunting): picked mon.noname-a con 0x7febcc1430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:56.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febcc142520 con 0x7febcc1430c0 2024-03-25T18:13:56.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febcc0b28a0 con 0x7febcc0a98f0 2024-03-25T18:13:56.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febcc0b2a30 con 0x7febcc13f710 2024-03-25T18:13:56.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:56.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.752+0000 7febd09b5640 10 monclient(hunting): authenticate will time out at 7192.005129s 2024-03-25T18:13:56.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.763+0000 7febcad9d640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7febcc1430c0 0x7febcc14dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:39806/0 (socket says 172.21.15.37:39806) 2024-03-25T18:13:56.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.763+0000 7febcad9d640 1 -- 172.21.15.37:0/1344055689 learned_addr learned my addr 172.21.15.37:0/1344055689 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:56.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.778+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097967141 0 0) 0x7febcc0b28a0 con 0x7febcc0a98f0 2024-03-25T18:13:56.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.778+0000 7febc9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:13:56.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.778+0000 7febc9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:56.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.778+0000 7febc9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:56.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.778+0000 7febc9d9b640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:13:56.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.778+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febb8003520 con 0x7febcc0a98f0 2024-03-25T18:13:56.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.781+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2502546676 0 0) 0x7febcc142520 con 0x7febcc1430c0 2024-03-25T18:13:56.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.781+0000 7febc9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:13:56.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.781+0000 7febc9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:56.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.781+0000 7febc9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:56.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.781+0000 7febc9d9b640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:13:56.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.781+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febcc0b28a0 con 0x7febcc1430c0 2024-03-25T18:13:56.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.787+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4145825375 0 0) 0x7febcc0b2a30 con 0x7febcc13f710 2024-03-25T18:13:56.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.787+0000 7febc9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:13:56.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.787+0000 7febc9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:56.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.787+0000 7febc9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:56.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.787+0000 7febc9d9b640 10 monclient(hunting): my global_id is 4499 2024-03-25T18:13:56.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.787+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febcc142520 con 0x7febcc13f710 2024-03-25T18:13:56.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3347108715 0 0) 0x7febcc0b28a0 con 0x7febcc1430c0 2024-03-25T18:13:56.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 >> v1:172.21.15.37:6790/0 conn(0x7febcc13f710 legacy=0x7febcc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:56.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 >> v1:172.21.15.117:6789/0 conn(0x7febcc0a98f0 legacy=0x7febcc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:56.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:13:56.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 1 monclient: found mon.noname-a 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febcc0b44c0 con 0x7febcc1430c0 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7febc40019f0 con 0x7febcc1430c0 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febc9d9b640 10 monclient: dump: 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:56.805 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:56.806 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:56.806 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:56.806 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:56.806 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:56.806 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:56.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febd09b5640 5 monclient: authenticate success, global_id 4503 2024-03-25T18:13:56.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.803+0000 7febd09b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:13:56.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.829+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febc4001b50 con 0x7febcc1430c0 2024-03-25T18:13:56.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.829+0000 7febc9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:56.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.829+0000 7febc9d9b640 1 -- 172.21.15.37:0/1344055689 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7febc4001f60 con 0x7febcc1430c0 2024-03-25T18:13:56.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.829+0000 7febc9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:56.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.829+0000 7febc9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.829+0000 7febc9d9b640 10 monclient: dump: 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.829+0000 7febd09b5640 10 monclient: get_monmap_and_config success 2024-03-25T18:13:56.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.829+0000 7febd09b5640 4 set_mon_vals no callback set 2024-03-25T18:13:56.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.830+0000 7febd09b5640 10 monclient: shutdown 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.830+0000 7febd09b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.830+0000 7febd09b5640 1 -- 172.21.15.37:0/1344055689 >> v1:172.21.15.37:6789/0 conn(0x7febcc1430c0 legacy=0x7febcc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.830+0000 7febd09b5640 1 -- 172.21.15.37:0/1344055689 shutdown_connections 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.830+0000 7febd09b5640 1 -- 172.21.15.37:0/1344055689 >> 172.21.15.37:0/1344055689 conn(0x7febcc0b7ea0 msgr2=0x7febcc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.830+0000 7febd09b5640 1 -- 172.21.15.37:0/1344055689 shutdown_connections 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.830+0000 7febd09b5640 1 -- 172.21.15.37:0/1344055689 wait complete. 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.831+0000 7febd09b5640 10 monclient: build_initial_monmap 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.831+0000 7febd09b5640 10 monclient: monmap: 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:56.833 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:13:56.834 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:13:56.834 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:13:56.834 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:13:56.834 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:56.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.831+0000 7febd09b5640 1 librados: starting msgr at 2024-03-25T18:13:56.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.831+0000 7febd09b5640 1 librados: starting objecter 2024-03-25T18:13:56.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.831+0000 7febd09b5640 1 Processor -- start 2024-03-25T18:13:56.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.831+0000 7febd09b5640 1 -- start start 2024-03-25T18:13:56.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.831+0000 7febd09b5640 1 librados: setting wanted keys 2024-03-25T18:13:56.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.831+0000 7febd09b5640 1 librados: calling monclient init 2024-03-25T18:13:56.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.831+0000 7febd09b5640 10 monclient: init 2024-03-25T18:13:56.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.832+0000 7febd09b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:56.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.832+0000 7febd09b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:56.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.832+0000 7febd09b5640 10 monclient(hunting): picked mon.noname-c con 0x7febcc0a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:13:56.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.832+0000 7febd09b5640 10 monclient(hunting): picked mon.noname-a con 0x7febcc13f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:13:56.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.833+0000 7febd09b5640 10 monclient(hunting): picked mon.noname-b con 0x7febcc1430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:13:56.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.833+0000 7febd09b5640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febcc1d48f0 con 0x7febcc13f710 2024-03-25T18:13:56.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.833+0000 7febd09b5640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febcc1397d0 con 0x7febcc1430c0 2024-03-25T18:13:56.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.833+0000 7febd09b5640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febcc1d4b50 con 0x7febcc0a98f0 2024-03-25T18:13:56.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.833+0000 7febd09b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:56.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.833+0000 7febd09b5640 10 monclient(hunting): authenticate will time out at 7192.085409s 2024-03-25T18:13:56.840 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.839+0000 7febcad9d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7febcc1430c0 0x7febcc1d7380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:34106/0 (socket says 172.21.15.37:34106) 2024-03-25T18:13:56.840 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.839+0000 7febcad9d640 1 -- 172.21.15.37:0/861743201 learned_addr learned my addr 172.21.15.37:0/861743201 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:13:56.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.843+0000 7febc2ffd640 10 client.?.objecter ms_handle_connect 0x7febcc0a98f0 2024-03-25T18:13:56.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.843+0000 7febc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:56.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.844+0000 7febc2ffd640 10 client.?.objecter ms_handle_connect 0x7febcc13f710 2024-03-25T18:13:56.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.844+0000 7febc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:56.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.846+0000 7febc2ffd640 10 client.?.objecter ms_handle_connect 0x7febcc1430c0 2024-03-25T18:13:56.847 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.846+0000 7febc2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:13:56.859 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 527304971 0 0) 0x7febcc1397d0 con 0x7febcc1430c0 2024-03-25T18:13:56.859 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:56.859 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:56.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:56.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:13:56.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febac004eb0 con 0x7febcc1430c0 2024-03-25T18:13:56.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244967700 0 0) 0x7febcc1d48f0 con 0x7febcc13f710 2024-03-25T18:13:56.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:56.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:56.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:56.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:13:56.860 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.858+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febcc1397d0 con 0x7febcc13f710 2024-03-25T18:13:56.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.863+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761152658 0 0) 0x7febcc1d4b50 con 0x7febcc0a98f0 2024-03-25T18:13:56.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.863+0000 7febc2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:13:56.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.863+0000 7febc2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:13:56.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.863+0000 7febc2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:13:56.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.863+0000 7febc2ffd640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:13:56.864 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.863+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febcc1d48f0 con 0x7febcc0a98f0 2024-03-25T18:13:56.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.879+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3386672459 0 0) 0x7febcc1397d0 con 0x7febcc13f710 2024-03-25T18:13:56.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.879+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febcc1d4b50 con 0x7febcc13f710 2024-03-25T18:13:56.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.879+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7febb4001870 con 0x7febcc13f710 2024-03-25T18:13:56.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.879+0000 7febc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:56.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.879+0000 7febc2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:13:56.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.879+0000 7febc2ffd640 10 monclient(hunting): dump: 2024-03-25T18:13:56.880 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:56.880 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:56.881 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:56.881 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:56.881 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:56.881 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:56.881 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:56.881 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:56.881 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:56.881 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:56.881 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.879+0000 7febc2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:13:56.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.885+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2775831875 0 0) 0x7febac004eb0 con 0x7febcc1430c0 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.885+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febcc1397d0 con 0x7febcc1430c0 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.885+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7febc4001f60 con 0x7febcc1430c0 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.885+0000 7febc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.885+0000 7febc2ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.885+0000 7febc2ffd640 10 monclient(hunting): dump: 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:56.887 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:56.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.886+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 367199715 0 0) 0x7febcc1d48f0 con 0x7febcc0a98f0 2024-03-25T18:13:56.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.886+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febac004eb0 con 0x7febcc0a98f0 2024-03-25T18:13:56.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.886+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7febbc0017b0 con 0x7febcc0a98f0 2024-03-25T18:13:56.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.886+0000 7febc2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:56.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.886+0000 7febc2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:13:56.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.886+0000 7febc2ffd640 10 monclient(hunting): dump: 2024-03-25T18:13:56.888 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:56.889 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:56.889 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:56.889 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:56.889 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:56.889 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:56.889 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:56.889 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:56.889 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:56.889 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:56.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2638596144 0 0) 0x7febcc1397d0 con 0x7febcc1430c0 2024-03-25T18:13:56.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 >> v1:172.21.15.37:6790/0 conn(0x7febcc0a98f0 legacy=0x7febcc1d41e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:56.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 >> v1:172.21.15.37:6789/0 conn(0x7febcc13f710 legacy=0x7febcc0aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:56.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febc2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:13:56.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febc2ffd640 1 monclient: found mon.b 2024-03-25T18:13:56.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febc2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:13:56.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febc2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:56.904 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febcc0b16b0 con 0x7febcc1430c0 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febc2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febc2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febc2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febd09b5640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febd09b5640 10 monclient: _renew_subs 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febd09b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febd09b5640 1 -- 172.21.15.37:0/861743201 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febcc0b0c10 con 0x7febcc1430c0 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febd09b5640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febd09b5640 10 monclient: _renew_subs 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febd09b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febd09b5640 1 -- 172.21.15.37:0/861743201 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febcc0b0610 con 0x7febcc1430c0 2024-03-25T18:13:56.905 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.903+0000 7febd09b5640 1 librados: init done 2024-03-25T18:13:56.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.904+0000 7febd09b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:56.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.904+0000 7febd09b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:56.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.904+0000 7febd09b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:56.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:56.904+0000 7febd09b5640 1 -- 172.21.15.37:0/861743201 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febcc1d4ce0 con 0x7febcc1430c0 2024-03-25T18:13:57.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.179+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febc4004e10 con 0x7febcc1430c0 2024-03-25T18:13:57.180 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.179+0000 7febc2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.179+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7febc4005120 con 0x7febcc1430c0 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.179+0000 7febc2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.179+0000 7febc2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.179+0000 7febc2ffd640 10 monclient: dump: 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:13:57.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.179+0000 7febc959a640 4 set_mon_vals no callback set 2024-03-25T18:13:57.182 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.180+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7febc401b0f0 con 0x7febcc1430c0 2024-03-25T18:13:57.182 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.180+0000 7febca59c640 10 monclient: get_auth_request con 0x7febac06e160 auth_method 0 2024-03-25T18:13:57.182 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.181+0000 7febc2ffd640 10 client.4507.objecter ms_handle_connect 0x7febac06e160 2024-03-25T18:13:57.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.185+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7febc4054ba0 con 0x7febcc1430c0 2024-03-25T18:13:57.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.185+0000 7febc2ffd640 10 client.4507.objecter ms_dispatch 0x7febcc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:13:57.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.185+0000 7febc2ffd640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:13:57.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.185+0000 7febc2ffd640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:13:57.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.185+0000 7febc2ffd640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-25T18:13:57.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.185+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febc40506b0 con 0x7febcc1430c0 2024-03-25T18:13:57.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.185+0000 7febc2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:13:57.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.185+0000 7febc2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:13:57.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.400+0000 7febd09b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:57.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.400+0000 7febd09b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:57.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.400+0000 7febd09b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:13:57.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.400+0000 7febd09b5640 1 -- 172.21.15.37:0/861743201 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febcc0627b0 con 0x7febcc1430c0 2024-03-25T18:13:57.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.438+0000 7febc2ffd640 1 -- 172.21.15.37:0/861743201 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7febc40525e0 con 0x7febcc1430c0 2024-03-25T18:13:57.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.438+0000 7febc2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:13:57.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.438+0000 7febc2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:13:57.440 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:13:57.440 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:13:57.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 10 librados: watch_flush enter 2024-03-25T18:13:57.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 10 librados: watch_flush exit 2024-03-25T18:13:57.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-25T18:13:57.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 10 client.4507.objecter successfully canceled tick 2024-03-25T18:13:57.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 1 -- 172.21.15.37:0/861743201 >> v1:172.21.15.117:6800/3703993921 conn(0x7febac06e160 legacy=0x7febac070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:57.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 10 monclient: shutdown 2024-03-25T18:13:57.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:13:57.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 1 -- 172.21.15.37:0/861743201 >> v1:172.21.15.117:6789/0 conn(0x7febcc1430c0 legacy=0x7febcc1d7380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:13:57.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 1 -- 172.21.15.37:0/861743201 shutdown_connections 2024-03-25T18:13:57.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 1 -- 172.21.15.37:0/861743201 >> 172.21.15.37:0/861743201 conn(0x7febcc0b7ea0 msgr2=0x7febcc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:13:57.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 1 -- 172.21.15.37:0/861743201 shutdown_connections 2024-03-25T18:13:57.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 1 -- 172.21.15.37:0/861743201 wait complete. 2024-03-25T18:13:57.443 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:13:57.441+0000 7febd09b5640 1 librados: shutdown 2024-03-25T18:13:57.455 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:04.262 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:04.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.379+0000 7f38287ed640 10 monclient: get_monmap_and_config 2024-03-25T18:14:04.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.380+0000 7f38287ed640 10 monclient: build_initial_monmap 2024-03-25T18:14:04.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.380+0000 7f38287ed640 10 monclient: monmap: 2024-03-25T18:14:04.381 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:04.381 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:04.381 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:14:04.380930+0000 2024-03-25T18:14:04.381 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:14:04.380930+0000 2024-03-25T18:14:04.381 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:04.381 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:04.381 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:04.382 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:14:04.382 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:14:04.382 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:04.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.380+0000 7f38287ed640 1 Processor -- start 2024-03-25T18:14:04.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.380+0000 7f38287ed640 1 -- start start 2024-03-25T18:14:04.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.380+0000 7f38287ed640 10 monclient: init 2024-03-25T18:14:04.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.381+0000 7f38287ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:04.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.381+0000 7f38287ed640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:04.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.381+0000 7f38287ed640 10 monclient(hunting): picked mon.noname-a con 0x7f38200a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:04.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.381+0000 7f38287ed640 10 monclient(hunting): picked mon.noname-b con 0x7f382013f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:04.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.381+0000 7f38287ed640 10 monclient(hunting): picked mon.noname-c con 0x7f38201430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:04.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.381+0000 7f38287ed640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3820142520 con 0x7f38200a98f0 2024-03-25T18:14:04.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.381+0000 7f38287ed640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38200b28a0 con 0x7f38201430c0 2024-03-25T18:14:04.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.381+0000 7f38287ed640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38200b2a30 con 0x7f382013f710 2024-03-25T18:14:04.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.381+0000 7f38287ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:04.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.381+0000 7f38287ed640 10 monclient(hunting): authenticate will time out at 7199.634210s 2024-03-25T18:14:04.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.382+0000 7f3825d88640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f382013f710 0x7f38201429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:40784/0 (socket says 172.21.15.37:40784) 2024-03-25T18:14:04.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.382+0000 7f3825d88640 1 -- 172.21.15.37:0/2703467557 learned_addr learned my addr 172.21.15.37:0/2703467557 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:04.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.392+0000 7f3825587640 1 -- 172.21.15.37:0/2703467557 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 58813386 0 0) 0x7f38200b2a30 con 0x7f382013f710 2024-03-25T18:14:04.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.392+0000 7f3825587640 10 monclient(hunting): _negotiate 2024-03-25T18:14:04.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.392+0000 7f3825587640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:04.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.392+0000 7f3825587640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:04.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.392+0000 7f3825587640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:14:04.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.392+0000 7f3825587640 1 -- 172.21.15.37:0/2703467557 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f380c003520 con 0x7f382013f710 2024-03-25T18:14:04.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 1 -- 172.21.15.37:0/2703467557 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1745228669 0 0) 0x7f380c003520 con 0x7f382013f710 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 1 -- 172.21.15.37:0/2703467557 >> v1:172.21.15.117:6789/0 conn(0x7f38201430c0 legacy=0x7f382014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 1 -- 172.21.15.37:0/2703467557 >> v1:172.21.15.37:6789/0 conn(0x7f38200a98f0 legacy=0x7f38200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 10 monclient: _finish_hunting 0 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 1 monclient: found mon.noname-b 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 1 -- 172.21.15.37:0/2703467557 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38200b44c0 con 0x7f382013f710 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 10 monclient: _finish_auth 0 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 10 monclient: _check_auth_tickets 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 1 -- 172.21.15.37:0/2703467557 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3810001690 con 0x7f382013f710 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f38287ed640 5 monclient: authenticate success, global_id 4505 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f38287ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:04.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f3825587640 10 monclient: dump: 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:04.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.402+0000 7f38287ed640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:04.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.414+0000 7f3825587640 1 -- 172.21.15.37:0/2703467557 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38100017f0 con 0x7f382013f710 2024-03-25T18:14:04.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.414+0000 7f3825587640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:04.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.414+0000 7f3825587640 1 -- 172.21.15.37:0/2703467557 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3810001c00 con 0x7f382013f710 2024-03-25T18:14:04.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.414+0000 7f3825587640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:04.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.414+0000 7f3825587640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:04.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.414+0000 7f3825587640 10 monclient: dump: 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.414+0000 7f38287ed640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:04.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.414+0000 7f38287ed640 4 set_mon_vals no callback set 2024-03-25T18:14:04.418 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.415+0000 7f38287ed640 10 monclient: shutdown 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.415+0000 7f38287ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.415+0000 7f38287ed640 1 -- 172.21.15.37:0/2703467557 >> v1:172.21.15.37:6790/0 conn(0x7f382013f710 legacy=0x7f38201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.415+0000 7f38287ed640 1 -- 172.21.15.37:0/2703467557 shutdown_connections 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.415+0000 7f38287ed640 1 -- 172.21.15.37:0/2703467557 >> 172.21.15.37:0/2703467557 conn(0x7f38200b7ea0 msgr2=0x7f38200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.415+0000 7f38287ed640 1 -- 172.21.15.37:0/2703467557 shutdown_connections 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.415+0000 7f38287ed640 1 -- 172.21.15.37:0/2703467557 wait complete. 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.415+0000 7f38287ed640 10 monclient: build_initial_monmap 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.415+0000 7f38287ed640 10 monclient: monmap: 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:04.419 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.416+0000 7f38287ed640 1 librados: starting msgr at 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.416+0000 7f38287ed640 1 librados: starting objecter 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.416+0000 7f38287ed640 1 Processor -- start 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.416+0000 7f38287ed640 1 -- start start 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.416+0000 7f38287ed640 1 librados: setting wanted keys 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.416+0000 7f38287ed640 1 librados: calling monclient init 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.416+0000 7f38287ed640 10 monclient: init 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.417+0000 7f38287ed640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.417+0000 7f38287ed640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.417+0000 7f38287ed640 10 monclient(hunting): picked mon.noname-c con 0x7f38200a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:04.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.417+0000 7f38287ed640 10 monclient(hunting): picked mon.noname-b con 0x7f382013f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:04.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.417+0000 7f38287ed640 10 monclient(hunting): picked mon.noname-a con 0x7f38201430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:04.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.417+0000 7f38287ed640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3820138c10 con 0x7f38201430c0 2024-03-25T18:14:04.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.417+0000 7f38287ed640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3820138e60 con 0x7f382013f710 2024-03-25T18:14:04.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.417+0000 7f38287ed640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3820139140 con 0x7f38200a98f0 2024-03-25T18:14:04.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.417+0000 7f38287ed640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:04.421 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.417+0000 7f38287ed640 10 monclient(hunting): authenticate will time out at 7199.669655s 2024-03-25T18:14:04.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.433+0000 7f3825d88640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f382013f710 0x7f3820132440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:38810/0 (socket says 172.21.15.37:38810) 2024-03-25T18:14:04.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.433+0000 7f3825d88640 1 -- 172.21.15.37:0/4117096430 learned_addr learned my addr 172.21.15.37:0/4117096430 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:04.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.435+0000 7f3816ffd640 10 client.?.objecter ms_handle_connect 0x7f382013f710 2024-03-25T18:14:04.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.435+0000 7f3816ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:04.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.439+0000 7f3816ffd640 10 client.?.objecter ms_handle_connect 0x7f38200a98f0 2024-03-25T18:14:04.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.439+0000 7f3816ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:04.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.440+0000 7f3816ffd640 10 client.?.objecter ms_handle_connect 0x7f38201430c0 2024-03-25T18:14:04.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.440+0000 7f3816ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:04.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.444+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3948932037 0 0) 0x7f3820138e60 con 0x7f382013f710 2024-03-25T18:14:04.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.444+0000 7f3816ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:04.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.444+0000 7f3816ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:04.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.444+0000 7f3816ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:04.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.444+0000 7f3816ffd640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:14:04.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.444+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3804004eb0 con 0x7f382013f710 2024-03-25T18:14:04.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.449+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2550872475 0 0) 0x7f3820139140 con 0x7f38200a98f0 2024-03-25T18:14:04.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.449+0000 7f3816ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:04.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.449+0000 7f3816ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:04.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.449+0000 7f3816ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:04.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.449+0000 7f3816ffd640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:14:04.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.449+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3820138e60 con 0x7f38200a98f0 2024-03-25T18:14:04.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.451+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249758289 0 0) 0x7f3820138c10 con 0x7f38201430c0 2024-03-25T18:14:04.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.451+0000 7f3816ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:04.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.451+0000 7f3816ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:04.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.451+0000 7f3816ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:04.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.451+0000 7f3816ffd640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:14:04.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.451+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3820139140 con 0x7f38201430c0 2024-03-25T18:14:04.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.463+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2757999990 0 0) 0x7f3804004eb0 con 0x7f382013f710 2024-03-25T18:14:04.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.463+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3820138c10 con 0x7f382013f710 2024-03-25T18:14:04.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.463+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3810001c00 con 0x7f382013f710 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.463+0000 7f3816ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.463+0000 7f3816ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.463+0000 7f3816ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:04.465 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.463+0000 7f3816ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.470+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2928975838 0 0) 0x7f3820139140 con 0x7f38201430c0 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.471+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3804004eb0 con 0x7f38201430c0 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.471+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3818001a90 con 0x7f38201430c0 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.471+0000 7f3816ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.471+0000 7f3816ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.471+0000 7f3816ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:04.472 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:04.473 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:04.473 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:04.473 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:04.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.479+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3586988026 0 0) 0x7f3820138e60 con 0x7f38200a98f0 2024-03-25T18:14:04.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.479+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3820139140 con 0x7f38200a98f0 2024-03-25T18:14:04.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.479+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f381c0018a0 con 0x7f38200a98f0 2024-03-25T18:14:04.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.479+0000 7f3816ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:04.480 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.479+0000 7f3816ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:04.481 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.479+0000 7f3816ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:04.481 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:04.481 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:04.481 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.481 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.481 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:04.481 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:04.481 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:04.481 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:04.481 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:04.481 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:04.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2189167734 0 0) 0x7f3820138c10 con 0x7f382013f710 2024-03-25T18:14:04.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 >> v1:172.21.15.37:6790/0 conn(0x7f38200a98f0 legacy=0x7f3820137bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:04.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 >> v1:172.21.15.37:6789/0 conn(0x7f38201430c0 legacy=0x7f38201e34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:04.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f3816ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:04.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f3816ffd640 1 monclient: found mon.b 2024-03-25T18:14:04.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f3816ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:04.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f3816ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:04.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3820139920 con 0x7f382013f710 2024-03-25T18:14:04.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f3816ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:04.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f3816ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:04.490 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f3816ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:04.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f38287ed640 5 monclient: authenticate success, global_id 4513 2024-03-25T18:14:04.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f38287ed640 10 monclient: _renew_subs 2024-03-25T18:14:04.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f38287ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:04.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f38287ed640 1 -- 172.21.15.37:0/4117096430 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38201382e0 con 0x7f382013f710 2024-03-25T18:14:04.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f38287ed640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:04.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f38287ed640 10 monclient: _renew_subs 2024-03-25T18:14:04.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f38287ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:04.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f38287ed640 1 -- 172.21.15.37:0/4117096430 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38200ab8b0 con 0x7f382013f710 2024-03-25T18:14:04.491 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.488+0000 7f38287ed640 1 librados: init done 2024-03-25T18:14:04.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.489+0000 7f38287ed640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:04.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.489+0000 7f38287ed640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:04.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.489+0000 7f38287ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:04.492 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.489+0000 7f38287ed640 1 -- 172.21.15.37:0/4117096430 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3820137a20 con 0x7f382013f710 2024-03-25T18:14:04.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.767+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3810004a90 con 0x7f382013f710 2024-03-25T18:14:04.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.767+0000 7f3816ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:04.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.767+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3810004e20 con 0x7f382013f710 2024-03-25T18:14:04.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.767+0000 7f3816ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:04.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.767+0000 7f3816ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:04.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.767+0000 7f3816ffd640 10 monclient: dump: 2024-03-25T18:14:04.768 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:04.768 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:04.768 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.768 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:04.768 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:04.769 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:04.769 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:04.769 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:04.769 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:04.769 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:04.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.767+0000 7f3824d86640 4 set_mon_vals no callback set 2024-03-25T18:14:04.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.767+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f3810005040 con 0x7f382013f710 2024-03-25T18:14:04.769 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.768+0000 7f3826589640 10 monclient: get_auth_request con 0x7f380406e0f0 auth_method 0 2024-03-25T18:14:04.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.768+0000 7f3816ffd640 10 client.4513.objecter ms_handle_connect 0x7f380406e0f0 2024-03-25T18:14:04.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.774+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f38100549b0 con 0x7f382013f710 2024-03-25T18:14:04.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.774+0000 7f3816ffd640 10 client.4513.objecter ms_dispatch 0x7f3820000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:04.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.774+0000 7f3816ffd640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:04.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.774+0000 7f3816ffd640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:04.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.774+0000 7f3816ffd640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T18:14:04.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.774+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38100504c0 con 0x7f382013f710 2024-03-25T18:14:04.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.775+0000 7f3816ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:04.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.775+0000 7f3816ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:05.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.999+0000 7f38287ed640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:05.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.999+0000 7f38287ed640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:05.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.999+0000 7f38287ed640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:05.001 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:04.999+0000 7f38287ed640 1 -- 172.21.15.37:0/4117096430 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f382013d010 con 0x7f382013f710 2024-03-25T18:14:05.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.031+0000 7f3816ffd640 1 -- 172.21.15.37:0/4117096430 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f38100523f0 con 0x7f382013f710 2024-03-25T18:14:05.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.031+0000 7f3816ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:05.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.031+0000 7f3816ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:05.032 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:14:05.032 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:05.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 10 librados: watch_flush enter 2024-03-25T18:14:05.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 10 librados: watch_flush exit 2024-03-25T18:14:05.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T18:14:05.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 10 client.4513.objecter successfully canceled tick 2024-03-25T18:14:05.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 1 -- 172.21.15.37:0/4117096430 >> v1:172.21.15.117:6800/3703993921 conn(0x7f380406e0f0 legacy=0x7f38040705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:05.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 10 monclient: shutdown 2024-03-25T18:14:05.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:05.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 1 -- 172.21.15.37:0/4117096430 >> v1:172.21.15.117:6789/0 conn(0x7f382013f710 legacy=0x7f3820132440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:05.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 1 -- 172.21.15.37:0/4117096430 shutdown_connections 2024-03-25T18:14:05.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 1 -- 172.21.15.37:0/4117096430 >> 172.21.15.37:0/4117096430 conn(0x7f38200b7ea0 msgr2=0x7f3820074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:05.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 1 -- 172.21.15.37:0/4117096430 shutdown_connections 2024-03-25T18:14:05.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 1 -- 172.21.15.37:0/4117096430 wait complete. 2024-03-25T18:14:05.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:05.033+0000 7f38287ed640 1 librados: shutdown 2024-03-25T18:14:05.046 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:10.363 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:06:45.708 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:10.369 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:06:45.714 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:10.408 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:06:45.753 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:10.422 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:06:45.767 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:10.425 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:06:45.738 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:10.429 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:06:45.742 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:10.429 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:06:45.775 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:10.453 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:06:45.766 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:10.455 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:06:45.801 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:11.853 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:11.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.969+0000 7f881c7f6640 10 monclient: get_monmap_and_config 2024-03-25T18:14:11.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.969+0000 7f881c7f6640 10 monclient: build_initial_monmap 2024-03-25T18:14:11.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.969+0000 7f881c7f6640 10 monclient: monmap: 2024-03-25T18:14:11.971 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:11.971 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:11.971 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:14:11.970671+0000 2024-03-25T18:14:11.971 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:14:11.970671+0000 2024-03-25T18:14:11.971 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:11.971 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:11.971 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:11.971 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:14:11.971 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:14:11.971 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:11.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.970+0000 7f881c7f6640 1 Processor -- start 2024-03-25T18:14:11.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.970+0000 7f881c7f6640 1 -- start start 2024-03-25T18:14:11.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.970+0000 7f881c7f6640 10 monclient: init 2024-03-25T18:14:11.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.971+0000 7f881c7f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:11.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.971+0000 7f881c7f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:11.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.971+0000 7f881c7f6640 10 monclient(hunting): picked mon.noname-a con 0x7f88140b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:11.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.971+0000 7f881c7f6640 10 monclient(hunting): picked mon.noname-b con 0x7f88140b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:11.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.971+0000 7f881c7f6640 10 monclient(hunting): picked mon.noname-c con 0x7f881414be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:11.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.971+0000 7f881c7f6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88140af940 con 0x7f88140b4d60 2024-03-25T18:14:11.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.971+0000 7f881c7f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88140b0a70 con 0x7f881414be90 2024-03-25T18:14:11.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.971+0000 7f881c7f6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88140b0d50 con 0x7f88140b1730 2024-03-25T18:14:11.974 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.971+0000 7f881c7f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:11.974 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.971+0000 7f881c7f6640 10 monclient(hunting): authenticate will time out at 7207.224128s 2024-03-25T18:14:11.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.982+0000 7f881ad93640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f881414be90 0x7f881414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:56548/0 (socket says 172.21.15.37:56548) 2024-03-25T18:14:11.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.982+0000 7f881ad93640 1 -- 172.21.15.37:0/3119853914 learned_addr learned my addr 172.21.15.37:0/3119853914 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:12.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.999+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2993959214 0 0) 0x7f88140b0d50 con 0x7f88140b1730 2024-03-25T18:14:12.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.999+0000 7f8819590640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.999+0000 7f8819590640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.999+0000 7f8819590640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.999+0000 7f8819590640 10 monclient(hunting): my global_id is 4511 2024-03-25T18:14:12.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:11.999+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8800003520 con 0x7f88140b1730 2024-03-25T18:14:12.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.002+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2014267288 0 0) 0x7f88140b0a70 con 0x7f881414be90 2024-03-25T18:14:12.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.002+0000 7f8819590640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.002+0000 7f8819590640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.002+0000 7f8819590640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.002+0000 7f8819590640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:14:12.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.002+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88140b0d50 con 0x7f881414be90 2024-03-25T18:14:12.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.004+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 66261112 0 0) 0x7f88140af940 con 0x7f88140b4d60 2024-03-25T18:14:12.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.004+0000 7f8819590640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.004+0000 7f8819590640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.004+0000 7f8819590640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.004+0000 7f8819590640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:14:12.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.004+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88140b0a70 con 0x7f88140b4d60 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2871871367 0 0) 0x7f8800003520 con 0x7f88140b1730 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 >> v1:172.21.15.117:6789/0 conn(0x7f881414be90 legacy=0x7f881414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 >> v1:172.21.15.37:6789/0 conn(0x7f88140b4d60 legacy=0x7f881413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 10 monclient: _finish_hunting 0 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 1 monclient: found mon.noname-b 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88140b1030 con 0x7f88140b1730 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 10 monclient: _finish_auth 0 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 10 monclient: _check_auth_tickets 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:12.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8804001690 con 0x7f88140b1730 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f8819590640 10 monclient: dump: 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f881c7f6640 5 monclient: authenticate success, global_id 4511 2024-03-25T18:14:12.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.012+0000 7f881c7f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.029+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88040017f0 con 0x7f88140b1730 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.029+0000 7f8819590640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.029+0000 7f8819590640 1 -- 172.21.15.37:0/3119853914 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8804001c00 con 0x7f88140b1730 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.029+0000 7f8819590640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.029+0000 7f8819590640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.029+0000 7f8819590640 10 monclient: dump: 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.029+0000 7f881c7f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:12.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.029+0000 7f881c7f6640 4 set_mon_vals no callback set 2024-03-25T18:14:12.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.030+0000 7f881c7f6640 10 monclient: shutdown 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.030+0000 7f881c7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.030+0000 7f881c7f6640 1 -- 172.21.15.37:0/3119853914 >> v1:172.21.15.37:6790/0 conn(0x7f88140b1730 legacy=0x7f88140b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.030+0000 7f881c7f6640 1 -- 172.21.15.37:0/3119853914 shutdown_connections 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.030+0000 7f881c7f6640 1 -- 172.21.15.37:0/3119853914 >> 172.21.15.37:0/3119853914 conn(0x7f88140617a0 msgr2=0x7f8814061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.030+0000 7f881c7f6640 1 -- 172.21.15.37:0/3119853914 shutdown_connections 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.030+0000 7f881c7f6640 1 -- 172.21.15.37:0/3119853914 wait complete. 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.031+0000 7f881c7f6640 10 monclient: build_initial_monmap 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.031+0000 7f881c7f6640 10 monclient: monmap: 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:12.033 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.031+0000 7f881c7f6640 1 librados: starting msgr at 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.031+0000 7f881c7f6640 1 librados: starting objecter 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.031+0000 7f881c7f6640 1 Processor -- start 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.032+0000 7f881c7f6640 1 -- start start 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.032+0000 7f881c7f6640 1 librados: setting wanted keys 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.032+0000 7f881c7f6640 1 librados: calling monclient init 2024-03-25T18:14:12.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.032+0000 7f881c7f6640 10 monclient: init 2024-03-25T18:14:12.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.032+0000 7f881c7f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:12.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.032+0000 7f881c7f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:12.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.032+0000 7f881c7f6640 10 monclient(hunting): picked mon.noname-a con 0x7f88140b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:12.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.032+0000 7f881c7f6640 10 monclient(hunting): picked mon.noname-c con 0x7f88140b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:12.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.032+0000 7f881c7f6640 10 monclient(hunting): picked mon.noname-b con 0x7f881414be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:12.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.033+0000 7f881c7f6640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88141433a0 con 0x7f88140b1730 2024-03-25T18:14:12.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.033+0000 7f881c7f6640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8814134a20 con 0x7f881414be90 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.033+0000 7f881c7f6640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f881413ae00 con 0x7f88140b4d60 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.033+0000 7f881c7f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:12.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.033+0000 7f881c7f6640 10 monclient(hunting): authenticate will time out at 7207.285327s 2024-03-25T18:14:12.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.047+0000 7f881ad93640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f881414be90 0x7f8814134310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:56592/0 (socket says 172.21.15.37:56592) 2024-03-25T18:14:12.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.047+0000 7f881ad93640 1 -- 172.21.15.37:0/812167250 learned_addr learned my addr 172.21.15.37:0/812167250 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:12.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.049+0000 7f880affd640 10 client.?.objecter ms_handle_connect 0x7f881414be90 2024-03-25T18:14:12.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.049+0000 7f880affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:12.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.052+0000 7f880affd640 10 client.?.objecter ms_handle_connect 0x7f88140b4d60 2024-03-25T18:14:12.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.052+0000 7f880affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:12.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.060+0000 7f880affd640 10 client.?.objecter ms_handle_connect 0x7f88140b1730 2024-03-25T18:14:12.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.060+0000 7f880affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:12.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.062+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4215842461 0 0) 0x7f8814134a20 con 0x7f881414be90 2024-03-25T18:14:12.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.062+0000 7f880affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.062+0000 7f880affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.062+0000 7f880affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.062+0000 7f880affd640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:14:12.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.062+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f87f8004eb0 con 0x7f881414be90 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.063+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3689966384 0 0) 0x7f88141433a0 con 0x7f88140b1730 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.063+0000 7f880affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.063+0000 7f880affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.063+0000 7f880affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.063+0000 7f880affd640 10 monclient(hunting): my global_id is 4518 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.063+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8814134a20 con 0x7f88140b1730 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.064+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215474844 0 0) 0x7f881413ae00 con 0x7f88140b4d60 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.064+0000 7f880affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.064+0000 7f880affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.064+0000 7f880affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.064+0000 7f880affd640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:14:12.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.064+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88141433a0 con 0x7f88140b4d60 2024-03-25T18:14:12.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1101926627 0 0) 0x7f8814134a20 con 0x7f88140b1730 2024-03-25T18:14:12.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f881413ae00 con 0x7f88140b1730 2024-03-25T18:14:12.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8810001870 con 0x7f88140b1730 2024-03-25T18:14:12.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:12.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 10 monclient(hunting): dump: 2024-03-25T18:14:12.083 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:12.083 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:12.083 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.083 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.083 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 548857287 0 0) 0x7f87f8004eb0 con 0x7f881414be90 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8814134a20 con 0x7f881414be90 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f880c001bf0 con 0x7f881414be90 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.081+0000 7f880affd640 10 monclient(hunting): dump: 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:12.084 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.082+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3359303095 0 0) 0x7f88141433a0 con 0x7f88140b4d60 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.082+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f87f8004eb0 con 0x7f88140b4d60 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.082+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8804001b00 con 0x7f88140b4d60 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.082+0000 7f880affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.082+0000 7f880affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.082+0000 7f880affd640 10 monclient(hunting): dump: 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:12.085 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:12.086 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.086 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.086 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.086 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:12.086 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:12.086 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:12.086 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:12.086 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:12.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3992009677 0 0) 0x7f8814134a20 con 0x7f881414be90 2024-03-25T18:14:12.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 >> v1:172.21.15.37:6790/0 conn(0x7f88140b4d60 legacy=0x7f8814133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 >> v1:172.21.15.37:6789/0 conn(0x7f88140b1730 legacy=0x7f881413b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f880affd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:12.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f880affd640 1 monclient: found mon.b 2024-03-25T18:14:12.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f880affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:12.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f880affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:12.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881413b0e0 con 0x7f881414be90 2024-03-25T18:14:12.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f880affd640 10 monclient: _finish_auth 0 2024-03-25T18:14:12.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f880affd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:12.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f880affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:12.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f881c7f6640 5 monclient: authenticate success, global_id 4519 2024-03-25T18:14:12.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f881c7f6640 10 monclient: _renew_subs 2024-03-25T18:14:12.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f881c7f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:12.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f881c7f6640 1 -- 172.21.15.37:0/812167250 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8814139bd0 con 0x7f881414be90 2024-03-25T18:14:12.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f881c7f6640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:12.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f881c7f6640 10 monclient: _renew_subs 2024-03-25T18:14:12.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f881c7f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:12.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f881c7f6640 1 -- 172.21.15.37:0/812167250 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8814142f50 con 0x7f881414be90 2024-03-25T18:14:12.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.091+0000 7f881c7f6640 1 librados: init done 2024-03-25T18:14:12.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.092+0000 7f881c7f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:12.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.092+0000 7f881c7f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:12.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.092+0000 7f881c7f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:12.095 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.092+0000 7f881c7f6640 1 -- 172.21.15.37:0/812167250 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8814140240 con 0x7f881414be90 2024-03-25T18:14:12.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.404+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f880c0020c0 con 0x7f881414be90 2024-03-25T18:14:12.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.404+0000 7f880affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:12.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.404+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f880c004d00 con 0x7f881414be90 2024-03-25T18:14:12.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.404+0000 7f880affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.404+0000 7f880affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:12.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.404+0000 7f880affd640 10 monclient: dump: 2024-03-25T18:14:12.406 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:12.406 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:12.406 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.406 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:12.406 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.407 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:12.407 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:12.407 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:12.407 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:12.407 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:12.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.404+0000 7f8818d8f640 4 set_mon_vals no callback set 2024-03-25T18:14:12.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.405+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f880c01ad50 con 0x7f881414be90 2024-03-25T18:14:12.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.406+0000 7f881a592640 10 monclient: get_auth_request con 0x7f87f806e160 auth_method 0 2024-03-25T18:14:12.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.406+0000 7f880affd640 10 client.4519.objecter ms_handle_connect 0x7f87f806e160 2024-03-25T18:14:12.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.412+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f880c054970 con 0x7f881414be90 2024-03-25T18:14:12.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.412+0000 7f880affd640 10 client.4519.objecter ms_dispatch 0x7f8814000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:12.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.412+0000 7f880affd640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:12.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.412+0000 7f880affd640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:12.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.412+0000 7f880affd640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-25T18:14:12.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.412+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f880c0502b0 con 0x7f881414be90 2024-03-25T18:14:12.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.412+0000 7f880affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:12.414 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.412+0000 7f880affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:12.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.653+0000 7f881c7f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:12.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.653+0000 7f881c7f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:12.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.653+0000 7f881c7f6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:12.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.653+0000 7f881c7f6640 1 -- 172.21.15.37:0/812167250 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8814136d90 con 0x7f881414be90 2024-03-25T18:14:12.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.682+0000 7f880affd640 1 -- 172.21.15.37:0/812167250 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f880c0523b0 con 0x7f881414be90 2024-03-25T18:14:12.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.682+0000 7f880affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:12.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.682+0000 7f880affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:12.683 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:14:12.683 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:12.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 10 librados: watch_flush enter 2024-03-25T18:14:12.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 10 librados: watch_flush exit 2024-03-25T18:14:12.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-25T18:14:12.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 10 client.4519.objecter successfully canceled tick 2024-03-25T18:14:12.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 1 -- 172.21.15.37:0/812167250 >> v1:172.21.15.117:6800/3703993921 conn(0x7f87f806e160 legacy=0x7f87f8070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 10 monclient: shutdown 2024-03-25T18:14:12.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:12.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 1 -- 172.21.15.37:0/812167250 >> v1:172.21.15.117:6789/0 conn(0x7f881414be90 legacy=0x7f8814134310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 1 -- 172.21.15.37:0/812167250 shutdown_connections 2024-03-25T18:14:12.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 1 -- 172.21.15.37:0/812167250 >> 172.21.15.37:0/812167250 conn(0x7f88140617a0 msgr2=0x7f8814137310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:12.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 1 -- 172.21.15.37:0/812167250 shutdown_connections 2024-03-25T18:14:12.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 1 -- 172.21.15.37:0/812167250 wait complete. 2024-03-25T18:14:12.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:12.685+0000 7f881c7f6640 1 librados: shutdown 2024-03-25T18:14:12.698 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:19.506 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:19.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.624+0000 7f12c950e640 10 monclient: get_monmap_and_config 2024-03-25T18:14:19.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.624+0000 7f12c950e640 10 monclient: build_initial_monmap 2024-03-25T18:14:19.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.624+0000 7f12c950e640 10 monclient: monmap: 2024-03-25T18:14:19.627 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:19.627 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:19.627 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:14:19.626217+0000 2024-03-25T18:14:19.627 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:14:19.626217+0000 2024-03-25T18:14:19.627 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:19.627 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:19.627 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:19.627 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:14:19.627 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:14:19.627 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:19.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 1 Processor -- start 2024-03-25T18:14:19.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 1 -- start start 2024-03-25T18:14:19.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 10 monclient: init 2024-03-25T18:14:19.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:19.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:19.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 10 monclient(hunting): picked mon.noname-b con 0x7f12c40a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:19.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 10 monclient(hunting): picked mon.noname-a con 0x7f12c413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:19.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 10 monclient(hunting): picked mon.noname-c con 0x7f12c41430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:19.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c4142520 con 0x7f12c413f710 2024-03-25T18:14:19.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c40b28a0 con 0x7f12c41430c0 2024-03-25T18:14:19.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c40b2a30 con 0x7f12c40a98f0 2024-03-25T18:14:19.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:19.629 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.625+0000 7f12c950e640 10 monclient(hunting): authenticate will time out at 7214.879278s 2024-03-25T18:14:19.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.637+0000 7f12c2ffd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f12c40a98f0 0x7f12c40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:50304/0 (socket says 172.21.15.37:50304) 2024-03-25T18:14:19.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.637+0000 7f12c2ffd640 1 -- 172.21.15.37:0/1117593509 learned_addr learned my addr 172.21.15.37:0/1117593509 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:19.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.650+0000 7f12c27fc640 1 -- 172.21.15.37:0/1117593509 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2665083996 0 0) 0x7f12c40b2a30 con 0x7f12c40a98f0 2024-03-25T18:14:19.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.650+0000 7f12c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:19.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.650+0000 7f12c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:19.652 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.650+0000 7f12c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:19.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.650+0000 7f12c27fc640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:14:19.653 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.650+0000 7f12c27fc640 1 -- 172.21.15.37:0/1117593509 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12b0003520 con 0x7f12c40a98f0 2024-03-25T18:14:19.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 1 -- 172.21.15.37:0/1117593509 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2327804118 0 0) 0x7f12c40b28a0 con 0x7f12c41430c0 2024-03-25T18:14:19.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:19.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:19.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:19.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:14:19.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 1 -- 172.21.15.37:0/1117593509 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c40b2a30 con 0x7f12c41430c0 2024-03-25T18:14:19.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 1 -- 172.21.15.37:0/1117593509 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2866032797 0 0) 0x7f12b0003520 con 0x7f12c40a98f0 2024-03-25T18:14:19.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 1 -- 172.21.15.37:0/1117593509 >> v1:172.21.15.117:6789/0 conn(0x7f12c41430c0 legacy=0x7f12c414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 1 -- 172.21.15.37:0/1117593509 >> v1:172.21.15.37:6789/0 conn(0x7f12c413f710 legacy=0x7f12c41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:19.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 1 monclient: found mon.noname-b 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 1 -- 172.21.15.37:0/1117593509 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c40b44c0 con 0x7f12c40a98f0 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 1 -- 172.21.15.37:0/1117593509 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f12b4001690 con 0x7f12c40a98f0 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c27fc640 10 monclient: dump: 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:19.667 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:19.668 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:19.668 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:19.668 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:19.668 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:19.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c950e640 5 monclient: authenticate success, global_id 4517 2024-03-25T18:14:19.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.663+0000 7f12c950e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:19.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.677+0000 7f12c27fc640 1 -- 172.21.15.37:0/1117593509 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12b4001870 con 0x7f12c40a98f0 2024-03-25T18:14:19.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.677+0000 7f12c27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:19.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.677+0000 7f12c27fc640 1 -- 172.21.15.37:0/1117593509 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f12b4001c80 con 0x7f12c40a98f0 2024-03-25T18:14:19.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.677+0000 7f12c27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:19.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.677+0000 7f12c27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:19.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.677+0000 7f12c27fc640 10 monclient: dump: 2024-03-25T18:14:19.679 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:19.680 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:19.680 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:19.680 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:19.680 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:19.680 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:19.680 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:19.680 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:19.680 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:19.680 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:19.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.677+0000 7f12c950e640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:19.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.677+0000 7f12c950e640 4 set_mon_vals no callback set 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.678+0000 7f12c950e640 10 monclient: shutdown 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.678+0000 7f12c950e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.678+0000 7f12c950e640 1 -- 172.21.15.37:0/1117593509 >> v1:172.21.15.37:6790/0 conn(0x7f12c40a98f0 legacy=0x7f12c40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.678+0000 7f12c950e640 1 -- 172.21.15.37:0/1117593509 shutdown_connections 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.678+0000 7f12c950e640 1 -- 172.21.15.37:0/1117593509 >> 172.21.15.37:0/1117593509 conn(0x7f12c40b7ea0 msgr2=0x7f12c40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.678+0000 7f12c950e640 1 -- 172.21.15.37:0/1117593509 shutdown_connections 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.678+0000 7f12c950e640 1 -- 172.21.15.37:0/1117593509 wait complete. 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.679+0000 7f12c950e640 10 monclient: build_initial_monmap 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.679+0000 7f12c950e640 10 monclient: monmap: 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:19.682 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.679+0000 7f12c950e640 1 librados: starting msgr at 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.679+0000 7f12c950e640 1 librados: starting objecter 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.679+0000 7f12c950e640 1 Processor -- start 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.679+0000 7f12c950e640 1 -- start start 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.679+0000 7f12c950e640 1 librados: setting wanted keys 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.679+0000 7f12c950e640 1 librados: calling monclient init 2024-03-25T18:14:19.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.679+0000 7f12c950e640 10 monclient: init 2024-03-25T18:14:19.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.680+0000 7f12c950e640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:19.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.680+0000 7f12c950e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:19.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.680+0000 7f12c950e640 10 monclient(hunting): picked mon.noname-b con 0x7f12c40a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:19.684 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.680+0000 7f12c950e640 10 monclient(hunting): picked mon.noname-c con 0x7f12c413f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:19.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.680+0000 7f12c950e640 10 monclient(hunting): picked mon.noname-a con 0x7f12c41430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:19.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.680+0000 7f12c950e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c4137590 con 0x7f12c41430c0 2024-03-25T18:14:19.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.680+0000 7f12c950e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c41db110 con 0x7f12c40a98f0 2024-03-25T18:14:19.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.680+0000 7f12c950e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12c41369c0 con 0x7f12c413f710 2024-03-25T18:14:19.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.680+0000 7f12c950e640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:19.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.680+0000 7f12c950e640 10 monclient(hunting): authenticate will time out at 7214.934178s 2024-03-25T18:14:19.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.683+0000 7f12c2ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f12c40a98f0 0x7f12c41daa00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:53818/0 (socket says 172.21.15.37:53818) 2024-03-25T18:14:19.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.683+0000 7f12c2ffd640 1 -- 172.21.15.37:0/1970081828 learned_addr learned my addr 172.21.15.37:0/1970081828 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:19.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.684+0000 7f12bb7fe640 10 client.?.objecter ms_handle_connect 0x7f12c40a98f0 2024-03-25T18:14:19.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.684+0000 7f12bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:19.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.697+0000 7f12bb7fe640 10 client.?.objecter ms_handle_connect 0x7f12c413f710 2024-03-25T18:14:19.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.697+0000 7f12bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:19.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.697+0000 7f12bb7fe640 10 client.?.objecter ms_handle_connect 0x7f12c41430c0 2024-03-25T18:14:19.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.697+0000 7f12bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:19.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.707+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3939815214 0 0) 0x7f12c41db110 con 0x7f12c40a98f0 2024-03-25T18:14:19.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.707+0000 7f12bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:19.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.707+0000 7f12bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:19.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.707+0000 7f12bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:19.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.707+0000 7f12bb7fe640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:14:19.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.708+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12a4004eb0 con 0x7f12c40a98f0 2024-03-25T18:14:19.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.712+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3196384654 0 0) 0x7f12c41369c0 con 0x7f12c413f710 2024-03-25T18:14:19.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.712+0000 7f12bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:19.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.712+0000 7f12bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:19.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.712+0000 7f12bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:19.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.712+0000 7f12bb7fe640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:14:19.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.712+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c41db110 con 0x7f12c413f710 2024-03-25T18:14:19.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.716+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1879657683 0 0) 0x7f12c4137590 con 0x7f12c41430c0 2024-03-25T18:14:19.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.716+0000 7f12bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:19.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.716+0000 7f12bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:19.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.716+0000 7f12bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:19.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.716+0000 7f12bb7fe640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:14:19.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.716+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f12c41369c0 con 0x7f12c41430c0 2024-03-25T18:14:19.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.724+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 621629419 0 0) 0x7f12a4004eb0 con 0x7f12c40a98f0 2024-03-25T18:14:19.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.724+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12c4137590 con 0x7f12c40a98f0 2024-03-25T18:14:19.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.724+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f12b4001bc0 con 0x7f12c40a98f0 2024-03-25T18:14:19.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.724+0000 7f12bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:19.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.724+0000 7f12bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:19.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.724+0000 7f12bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:19.726 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:19.726 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:19.726 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:19.727 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:19.727 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:19.727 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:19.727 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:19.727 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:19.727 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:19.727 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:19.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.724+0000 7f12bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:19.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.731+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1820136157 0 0) 0x7f12c41369c0 con 0x7f12c41430c0 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.731+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f12a4004eb0 con 0x7f12c41430c0 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.731+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f12bc001bd0 con 0x7f12c41430c0 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.731+0000 7f12bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.731+0000 7f12bb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.731+0000 7f12bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:19.734 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:19.735 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:19.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.741+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1357954363 0 0) 0x7f12a4004eb0 con 0x7f12c41430c0 2024-03-25T18:14:19.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.741+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 >> v1:172.21.15.37:6790/0 conn(0x7f12c413f710 legacy=0x7f12c40b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.741+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 >> v1:172.21.15.117:6789/0 conn(0x7f12c40a98f0 legacy=0x7f12c41daa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.741+0000 7f12bb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:19.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.741+0000 7f12bb7fe640 1 monclient: found mon.a 2024-03-25T18:14:19.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.741+0000 7f12bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:19.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12bb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:19.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12c4136b50 con 0x7f12c41430c0 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12bb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12bb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12c950e640 5 monclient: authenticate success, global_id 4524 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12c950e640 10 monclient: _renew_subs 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12c950e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12c950e640 1 -- 172.21.15.37:0/1970081828 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12c4137a70 con 0x7f12c41430c0 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12c950e640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12c950e640 10 monclient: _renew_subs 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12c950e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12c950e640 1 -- 172.21.15.37:0/1970081828 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f12c413b970 con 0x7f12c41430c0 2024-03-25T18:14:19.745 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12c950e640 1 librados: init done 2024-03-25T18:14:19.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.742+0000 7f12c950e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:19.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.743+0000 7f12c950e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:19.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.743+0000 7f12c950e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:19.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:19.743+0000 7f12c950e640 1 -- 172.21.15.37:0/1970081828 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12c40627b0 con 0x7f12c41430c0 2024-03-25T18:14:20.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.086+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f12bc001310 con 0x7f12c41430c0 2024-03-25T18:14:20.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.086+0000 7f12bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:20.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.086+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f12bc0020e0 con 0x7f12c41430c0 2024-03-25T18:14:20.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.086+0000 7f12bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:20.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.086+0000 7f12bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:20.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.086+0000 7f12bb7fe640 10 monclient: dump: 2024-03-25T18:14:20.089 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:20.089 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:20.089 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:20.089 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:20.090 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:20.090 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:20.090 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:20.090 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:20.090 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:20.090 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:20.090 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.086+0000 7f12c1ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:20.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.087+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f12bc019a40 con 0x7f12c41430c0 2024-03-25T18:14:20.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.088+0000 7f12c2ffd640 10 monclient: get_auth_request con 0x7f12a406dfb0 auth_method 0 2024-03-25T18:14:20.091 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.088+0000 7f12bb7fe640 10 client.4524.objecter ms_handle_connect 0x7f12a406dfb0 2024-03-25T18:14:20.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.094+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f12bc053770 con 0x7f12c41430c0 2024-03-25T18:14:20.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.094+0000 7f12bb7fe640 10 client.4524.objecter ms_dispatch 0x7f12c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:20.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.094+0000 7f12bb7fe640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:20.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.094+0000 7f12bb7fe640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:20.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.094+0000 7f12bb7fe640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-25T18:14:20.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.094+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f12bc04f0b0 con 0x7f12c41430c0 2024-03-25T18:14:20.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.094+0000 7f12bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:20.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.094+0000 7f12bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:20.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.330+0000 7f12c950e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:20.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.330+0000 7f12c950e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:20.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.330+0000 7f12c950e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:20.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.330+0000 7f12c950e640 1 -- 172.21.15.37:0/1970081828 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f12c4069f40 con 0x7f12c41430c0 2024-03-25T18:14:20.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.381+0000 7f12bb7fe640 1 -- 172.21.15.37:0/1970081828 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f12bc0511b0 con 0x7f12c41430c0 2024-03-25T18:14:20.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.381+0000 7f12bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:20.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.381+0000 7f12bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:20.383 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:14:20.383 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:20.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 10 librados: watch_flush enter 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 10 librados: watch_flush exit 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 10 client.4524.objecter successfully canceled tick 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 1 -- 172.21.15.37:0/1970081828 >> v1:172.21.15.117:6800/3703993921 conn(0x7f12a406dfb0 legacy=0x7f12a4070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 10 monclient: shutdown 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 1 -- 172.21.15.37:0/1970081828 >> v1:172.21.15.37:6789/0 conn(0x7f12c41430c0 legacy=0x7f12c41e2d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 1 -- 172.21.15.37:0/1970081828 shutdown_connections 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 1 -- 172.21.15.37:0/1970081828 >> 172.21.15.37:0/1970081828 conn(0x7f12c40b7ea0 msgr2=0x7f12c413d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 1 -- 172.21.15.37:0/1970081828 shutdown_connections 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 1 -- 172.21.15.37:0/1970081828 wait complete. 2024-03-25T18:14:20.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:20.383+0000 7f12c950e640 1 librados: shutdown 2024-03-25T18:14:20.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:27.204 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:27.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.320+0000 7f0515286640 10 monclient: get_monmap_and_config 2024-03-25T18:14:27.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.320+0000 7f0515286640 10 monclient: build_initial_monmap 2024-03-25T18:14:27.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.320+0000 7f0515286640 10 monclient: monmap: 2024-03-25T18:14:27.323 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:14:27.322711+0000 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:14:27.322711+0000 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.321+0000 7f0515286640 1 Processor -- start 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.321+0000 7f0515286640 1 -- start start 2024-03-25T18:14:27.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.321+0000 7f0515286640 10 monclient: init 2024-03-25T18:14:27.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.322+0000 7f0515286640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:27.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.322+0000 7f0515286640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:27.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.322+0000 7f0515286640 10 monclient(hunting): picked mon.noname-a con 0x7f05100a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:27.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.322+0000 7f0515286640 10 monclient(hunting): picked mon.noname-c con 0x7f051013f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:27.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.322+0000 7f0515286640 10 monclient(hunting): picked mon.noname-b con 0x7f05101430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:27.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.322+0000 7f0515286640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0510142520 con 0x7f05100a98f0 2024-03-25T18:14:27.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.322+0000 7f0515286640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05100b28a0 con 0x7f051013f710 2024-03-25T18:14:27.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.322+0000 7f0515286640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05100b2a30 con 0x7f05101430c0 2024-03-25T18:14:27.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.322+0000 7f0515286640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:27.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.322+0000 7f0515286640 10 monclient(hunting): authenticate will time out at 7222.575480s 2024-03-25T18:14:27.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.335+0000 7f050e7fc640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f051013f710 0x7f05101429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:58292/0 (socket says 172.21.15.37:58292) 2024-03-25T18:14:27.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.335+0000 7f050e7fc640 1 -- 172.21.15.37:0/1733750990 learned_addr learned my addr 172.21.15.37:0/1733750990 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.349+0000 7f050dffb640 1 -- 172.21.15.37:0/1733750990 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4205293246 0 0) 0x7f05100b28a0 con 0x7f051013f710 2024-03-25T18:14:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.349+0000 7f050dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.349+0000 7f050dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.349+0000 7f050dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.349+0000 7f050dffb640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:14:27.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.349+0000 7f050dffb640 1 -- 172.21.15.37:0/1733750990 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04f8003520 con 0x7f051013f710 2024-03-25T18:14:27.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.354+0000 7f050dffb640 1 -- 172.21.15.37:0/1733750990 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3775722274 0 0) 0x7f0510142520 con 0x7f05100a98f0 2024-03-25T18:14:27.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.354+0000 7f050dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:27.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.354+0000 7f050dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:27.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.354+0000 7f050dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:27.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.354+0000 7f050dffb640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:14:27.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.354+0000 7f050dffb640 1 -- 172.21.15.37:0/1733750990 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05100b28a0 con 0x7f05100a98f0 2024-03-25T18:14:27.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 1 -- 172.21.15.37:0/1733750990 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2826717613 0 0) 0x7f05100b28a0 con 0x7f05100a98f0 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 1 -- 172.21.15.37:0/1733750990 >> v1:172.21.15.37:6790/0 conn(0x7f05101430c0 legacy=0x7f051014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 1 -- 172.21.15.37:0/1733750990 >> v1:172.21.15.117:6789/0 conn(0x7f051013f710 legacy=0x7f05101429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 1 monclient: found mon.noname-a 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 1 -- 172.21.15.37:0/1733750990 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05100b44c0 con 0x7f05100a98f0 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 1 -- 172.21.15.37:0/1733750990 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0500001670 con 0x7f05100a98f0 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:27.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f050dffb640 10 monclient: dump: 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f0515286640 5 monclient: authenticate success, global_id 4527 2024-03-25T18:14:27.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.359+0000 7f0515286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:27.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f050dffb640 1 -- 172.21.15.37:0/1733750990 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f05000017d0 con 0x7f05100a98f0 2024-03-25T18:14:27.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f050dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:27.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f050dffb640 1 -- 172.21.15.37:0/1733750990 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0500001c20 con 0x7f05100a98f0 2024-03-25T18:14:27.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f050dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:27.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f050dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:27.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f050dffb640 10 monclient: dump: 2024-03-25T18:14:27.375 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:27.375 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:27.375 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:27.376 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:27.376 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:27.376 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:27.376 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:27.376 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:27.376 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:27.376 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:27.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f0515286640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:27.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f0515286640 4 set_mon_vals no callback set 2024-03-25T18:14:27.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f0515286640 10 monclient: shutdown 2024-03-25T18:14:27.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f0515286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:27.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f0515286640 1 -- 172.21.15.37:0/1733750990 >> v1:172.21.15.37:6789/0 conn(0x7f05100a98f0 legacy=0x7f05100a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f0515286640 1 -- 172.21.15.37:0/1733750990 shutdown_connections 2024-03-25T18:14:27.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f0515286640 1 -- 172.21.15.37:0/1733750990 >> 172.21.15.37:0/1733750990 conn(0x7f05100b7ea0 msgr2=0x7f05100b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:27.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f0515286640 1 -- 172.21.15.37:0/1733750990 shutdown_connections 2024-03-25T18:14:27.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.373+0000 7f0515286640 1 -- 172.21.15.37:0/1733750990 wait complete. 2024-03-25T18:14:27.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.374+0000 7f0515286640 10 monclient: build_initial_monmap 2024-03-25T18:14:27.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.374+0000 7f0515286640 10 monclient: monmap: 2024-03-25T18:14:27.379 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:27.379 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.374+0000 7f0515286640 1 librados: starting msgr at 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.374+0000 7f0515286640 1 librados: starting objecter 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.374+0000 7f0515286640 1 Processor -- start 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.374+0000 7f0515286640 1 -- start start 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.374+0000 7f0515286640 1 librados: setting wanted keys 2024-03-25T18:14:27.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.374+0000 7f0515286640 1 librados: calling monclient init 2024-03-25T18:14:27.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.374+0000 7f0515286640 10 monclient: init 2024-03-25T18:14:27.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.375+0000 7f0515286640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:27.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.375+0000 7f0515286640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:27.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.375+0000 7f0515286640 10 monclient(hunting): picked mon.noname-b con 0x7f05100a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:27.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.375+0000 7f0515286640 10 monclient(hunting): picked mon.noname-c con 0x7f051013f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:27.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.375+0000 7f0515286640 10 monclient(hunting): picked mon.noname-a con 0x7f05101430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:27.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.375+0000 7f0515286640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05101da8f0 con 0x7f05101430c0 2024-03-25T18:14:27.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.375+0000 7f0515286640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05101399f0 con 0x7f05100a98f0 2024-03-25T18:14:27.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.375+0000 7f0515286640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05101db770 con 0x7f051013f710 2024-03-25T18:14:27.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.375+0000 7f0515286640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:27.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.375+0000 7f0515286640 10 monclient(hunting): authenticate will time out at 7222.629259s 2024-03-25T18:14:27.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.383+0000 7f050f7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f05101430c0 0x7f05101e1c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:53692/0 (socket says 172.21.15.37:53692) 2024-03-25T18:14:27.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.383+0000 7f050f7fe640 1 -- 172.21.15.37:0/2299585617 learned_addr learned my addr 172.21.15.37:0/2299585617 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:27.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.388+0000 7f04f77fe640 10 client.?.objecter ms_handle_connect 0x7f05101430c0 2024-03-25T18:14:27.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.388+0000 7f04f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:27.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.398+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3568756864 0 0) 0x7f05101da8f0 con 0x7f05101430c0 2024-03-25T18:14:27.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.398+0000 7f04f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:27.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.398+0000 7f04f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:27.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.398+0000 7f04f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:27.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.398+0000 7f04f77fe640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:14:27.400 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.398+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc004eb0 con 0x7f05101430c0 2024-03-25T18:14:27.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.400+0000 7f04f77fe640 10 client.?.objecter ms_handle_connect 0x7f05100a98f0 2024-03-25T18:14:27.402 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.400+0000 7f04f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:27.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.406+0000 7f04f77fe640 10 client.?.objecter ms_handle_connect 0x7f051013f710 2024-03-25T18:14:27.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.406+0000 7f04f77fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:27.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.407+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3351908789 0 0) 0x7f05101399f0 con 0x7f05100a98f0 2024-03-25T18:14:27.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.407+0000 7f04f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:27.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.407+0000 7f04f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:27.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.407+0000 7f04f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:27.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.407+0000 7f04f77fe640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:14:27.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.407+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f05101da8f0 con 0x7f05100a98f0 2024-03-25T18:14:27.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.415+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2729708931 0 0) 0x7f04fc004eb0 con 0x7f05101430c0 2024-03-25T18:14:27.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.415+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05101399f0 con 0x7f05101430c0 2024-03-25T18:14:27.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.415+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0508001c30 con 0x7f05101430c0 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.415+0000 7f04f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.415+0000 7f04f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.415+0000 7f04f77fe640 10 monclient(hunting): dump: 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:27.418 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:27.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.415+0000 7f04f77fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:27.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.423+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360141818 0 0) 0x7f05101db770 con 0x7f051013f710 2024-03-25T18:14:27.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.423+0000 7f04f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:27.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.423+0000 7f04f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:27.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.423+0000 7f04f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:27.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.423+0000 7f04f77fe640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:14:27.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.423+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04fc004eb0 con 0x7f051013f710 2024-03-25T18:14:27.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1388953339 0 0) 0x7f05101399f0 con 0x7f05101430c0 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 >> v1:172.21.15.37:6790/0 conn(0x7f051013f710 legacy=0x7f05101381f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 >> v1:172.21.15.117:6789/0 conn(0x7f05100a98f0 legacy=0x7f05101db060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f04f77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f04f77fe640 1 monclient: found mon.a 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f04f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f04f77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05101dab40 con 0x7f05101430c0 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f04f77fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f04f77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f04f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f0515286640 5 monclient: authenticate success, global_id 4530 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f0515286640 10 monclient: _renew_subs 2024-03-25T18:14:27.431 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f0515286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:27.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f0515286640 1 -- 172.21.15.37:0/2299585617 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0510137870 con 0x7f05101430c0 2024-03-25T18:14:27.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f0515286640 10 client.4530.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:27.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f0515286640 10 monclient: _renew_subs 2024-03-25T18:14:27.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f0515286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:27.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f0515286640 1 -- 172.21.15.37:0/2299585617 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0510138ae0 con 0x7f05101430c0 2024-03-25T18:14:27.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.429+0000 7f0515286640 1 librados: init done 2024-03-25T18:14:27.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.430+0000 7f0515286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:27.433 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.430+0000 7f0515286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:27.433 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.430+0000 7f0515286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:27.433 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.430+0000 7f0515286640 1 -- 172.21.15.37:0/2299585617 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05100627b0 con 0x7f05101430c0 2024-03-25T18:14:27.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.745+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0508003990 con 0x7f05101430c0 2024-03-25T18:14:27.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.745+0000 7f04f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:27.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.745+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0508003da0 con 0x7f05101430c0 2024-03-25T18:14:27.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.745+0000 7f04f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:27.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.745+0000 7f04f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:27.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.745+0000 7f04f77fe640 10 monclient: dump: 2024-03-25T18:14:27.747 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:27.748 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:27.748 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:27.748 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:27.748 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:27.748 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:27.748 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:27.748 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:27.748 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:27.748 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:27.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.746+0000 7f050d7fa640 4 set_mon_vals no callback set 2024-03-25T18:14:27.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.746+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f0508003fc0 con 0x7f05101430c0 2024-03-25T18:14:27.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.747+0000 7f050effd640 10 monclient: get_auth_request con 0x7f04fc06de50 auth_method 0 2024-03-25T18:14:27.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.747+0000 7f04f77fe640 10 client.4530.objecter ms_handle_connect 0x7f04fc06de50 2024-03-25T18:14:27.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.754+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f0508054a30 con 0x7f05101430c0 2024-03-25T18:14:27.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.754+0000 7f04f77fe640 10 client.4530.objecter ms_dispatch 0x7f0510000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:27.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.754+0000 7f04f77fe640 3 client.4530.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:27.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.754+0000 7f04f77fe640 3 client.4530.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:27.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.755+0000 7f04f77fe640 20 client.4530.objecter dump_active .. 0 homeless 2024-03-25T18:14:27.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.755+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0508050540 con 0x7f05101430c0 2024-03-25T18:14:27.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.755+0000 7f04f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:27.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:27.755+0000 7f04f77fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:28.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.015+0000 7f0515286640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:28.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.015+0000 7f0515286640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:28.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.015+0000 7f0515286640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:28.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.015+0000 7f0515286640 1 -- 172.21.15.37:0/2299585617 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0510069f40 con 0x7f05101430c0 2024-03-25T18:14:28.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.042+0000 7f04f77fe640 1 -- 172.21.15.37:0/2299585617 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f0508052470 con 0x7f05101430c0 2024-03-25T18:14:28.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.042+0000 7f04f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:28.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.042+0000 7f04f77fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:28.044 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:14:28.044 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:28.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.044+0000 7f0515286640 10 librados: watch_flush enter 2024-03-25T18:14:28.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.044+0000 7f0515286640 10 librados: watch_flush exit 2024-03-25T18:14:28.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.044+0000 7f0515286640 20 client.4530.objecter shutdown clearing up homeless session... 2024-03-25T18:14:28.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.044+0000 7f0515286640 10 client.4530.objecter successfully canceled tick 2024-03-25T18:14:28.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.044+0000 7f0515286640 1 -- 172.21.15.37:0/2299585617 >> v1:172.21.15.117:6800/3703993921 conn(0x7f04fc06de50 legacy=0x7f04fc070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:28.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.044+0000 7f0515286640 10 monclient: shutdown 2024-03-25T18:14:28.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.044+0000 7f0515286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:28.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.044+0000 7f0515286640 1 -- 172.21.15.37:0/2299585617 >> v1:172.21.15.37:6789/0 conn(0x7f05101430c0 legacy=0x7f05101e1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:28.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.044+0000 7f0515286640 1 -- 172.21.15.37:0/2299585617 shutdown_connections 2024-03-25T18:14:28.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.044+0000 7f0515286640 1 -- 172.21.15.37:0/2299585617 >> 172.21.15.37:0/2299585617 conn(0x7f05100b7ea0 msgr2=0x7f051013d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:28.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.045+0000 7f0515286640 1 -- 172.21.15.37:0/2299585617 shutdown_connections 2024-03-25T18:14:28.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.045+0000 7f0515286640 1 -- 172.21.15.37:0/2299585617 wait complete. 2024-03-25T18:14:28.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:28.045+0000 7f0515286640 1 librados: shutdown 2024-03-25T18:14:28.058 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:30.396 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:07:05.742 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:30.406 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:07:05.751 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:30.426 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:07:05.772 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:30.443 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:07:05.788 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:30.447 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:07:05.792 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:30.476 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:07:05.789 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:30.483 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:07:05.796 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:30.488 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:07:05.834 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:30.520 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:07:05.833 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:34.864 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:34.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.981+0000 7fbf63474640 10 monclient: get_monmap_and_config 2024-03-25T18:14:34.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.981+0000 7fbf63474640 10 monclient: build_initial_monmap 2024-03-25T18:14:34.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.981+0000 7fbf63474640 10 monclient: monmap: 2024-03-25T18:14:34.982 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:34.982 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:34.983 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:14:34.982251+0000 2024-03-25T18:14:34.983 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:14:34.982251+0000 2024-03-25T18:14:34.983 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:34.983 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:34.983 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:34.983 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:14:34.983 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:14:34.983 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:34.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.982+0000 7fbf63474640 1 Processor -- start 2024-03-25T18:14:34.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.982+0000 7fbf63474640 1 -- start start 2024-03-25T18:14:34.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.982+0000 7fbf63474640 10 monclient: init 2024-03-25T18:14:34.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.983+0000 7fbf63474640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:34.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.983+0000 7fbf63474640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:34.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.983+0000 7fbf63474640 10 monclient(hunting): picked mon.noname-c con 0x7fbf5c0a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:34.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.983+0000 7fbf63474640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5c13f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:34.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.983+0000 7fbf63474640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5c1430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:34.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.983+0000 7fbf63474640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c142520 con 0x7fbf5c13f710 2024-03-25T18:14:34.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.983+0000 7fbf63474640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c0b28a0 con 0x7fbf5c0a98f0 2024-03-25T18:14:34.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.983+0000 7fbf63474640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c0b2a30 con 0x7fbf5c1430c0 2024-03-25T18:14:34.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.983+0000 7fbf63474640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:34.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.983+0000 7fbf63474640 10 monclient(hunting): authenticate will time out at 7230.235425s 2024-03-25T18:14:34.992 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.991+0000 7fbf61210640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fbf5c0a98f0 0x7fbf5c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:36428/0 (socket says 172.21.15.37:36428) 2024-03-25T18:14:34.992 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:34.991+0000 7fbf61210640 1 -- 172.21.15.37:0/2864244952 learned_addr learned my addr 172.21.15.37:0/2864244952 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:35.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.005+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3280135798 0 0) 0x7fbf5c0b28a0 con 0x7fbf5c0a98f0 2024-03-25T18:14:35.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.005+0000 7fbf53fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:35.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.005+0000 7fbf53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:35.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.005+0000 7fbf53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:35.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.005+0000 7fbf53fff640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:14:35.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.005+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf48003520 con 0x7fbf5c0a98f0 2024-03-25T18:14:35.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.009+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1702691402 0 0) 0x7fbf5c142520 con 0x7fbf5c13f710 2024-03-25T18:14:35.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.009+0000 7fbf53fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:35.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.009+0000 7fbf53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:35.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.009+0000 7fbf53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:35.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.009+0000 7fbf53fff640 10 monclient(hunting): my global_id is 4533 2024-03-25T18:14:35.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.009+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c0b28a0 con 0x7fbf5c13f710 2024-03-25T18:14:35.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.018+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708111905 0 0) 0x7fbf5c0b2a30 con 0x7fbf5c1430c0 2024-03-25T18:14:35.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.018+0000 7fbf53fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:35.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.018+0000 7fbf53fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:35.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.018+0000 7fbf53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:35.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.018+0000 7fbf53fff640 10 monclient(hunting): my global_id is 4529 2024-03-25T18:14:35.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.018+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c142520 con 0x7fbf5c1430c0 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3716805379 0 0) 0x7fbf48003520 con 0x7fbf5c0a98f0 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 >> v1:172.21.15.37:6790/0 conn(0x7fbf5c1430c0 legacy=0x7fbf5c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 >> v1:172.21.15.37:6789/0 conn(0x7fbf5c13f710 legacy=0x7fbf5c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 1 monclient: found mon.noname-c 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c0b44c0 con 0x7fbf5c0a98f0 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbf58001690 con 0x7fbf5c0a98f0 2024-03-25T18:14:35.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf63474640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf63474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf53fff640 10 monclient: dump: 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:35.022 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:35.023 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.019+0000 7fbf63474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:35.040 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.039+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf580017f0 con 0x7fbf5c0a98f0 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.039+0000 7fbf53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.039+0000 7fbf53fff640 1 -- 172.21.15.37:0/2864244952 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbf58001c00 con 0x7fbf5c0a98f0 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.039+0000 7fbf63474640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.039+0000 7fbf63474640 4 set_mon_vals no callback set 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.039+0000 7fbf53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.039+0000 7fbf53fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.039+0000 7fbf53fff640 10 monclient: dump: 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:35.041 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:35.042 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:35.042 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:35.042 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:35.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.040+0000 7fbf63474640 10 monclient: shutdown 2024-03-25T18:14:35.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.040+0000 7fbf63474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:35.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.040+0000 7fbf63474640 1 -- 172.21.15.37:0/2864244952 >> v1:172.21.15.117:6789/0 conn(0x7fbf5c0a98f0 legacy=0x7fbf5c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:35.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.040+0000 7fbf63474640 1 -- 172.21.15.37:0/2864244952 shutdown_connections 2024-03-25T18:14:35.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.040+0000 7fbf63474640 1 -- 172.21.15.37:0/2864244952 >> 172.21.15.37:0/2864244952 conn(0x7fbf5c0b7ea0 msgr2=0x7fbf5c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:35.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.040+0000 7fbf63474640 1 -- 172.21.15.37:0/2864244952 shutdown_connections 2024-03-25T18:14:35.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.040+0000 7fbf63474640 1 -- 172.21.15.37:0/2864244952 wait complete. 2024-03-25T18:14:35.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.041+0000 7fbf63474640 10 monclient: build_initial_monmap 2024-03-25T18:14:35.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.041+0000 7fbf63474640 10 monclient: monmap: 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 1 librados: starting msgr at 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 1 librados: starting objecter 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 1 Processor -- start 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 1 -- start start 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 1 librados: setting wanted keys 2024-03-25T18:14:35.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 1 librados: calling monclient init 2024-03-25T18:14:35.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 10 monclient: init 2024-03-25T18:14:35.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:35.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:35.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 10 monclient(hunting): picked mon.noname-c con 0x7fbf5c0a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:35.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 10 monclient(hunting): picked mon.noname-b con 0x7fbf5c13f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:35.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 10 monclient(hunting): picked mon.noname-a con 0x7fbf5c1430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:35.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.042+0000 7fbf63474640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c138a50 con 0x7fbf5c1430c0 2024-03-25T18:14:35.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.043+0000 7fbf63474640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c0ab9e0 con 0x7fbf5c13f710 2024-03-25T18:14:35.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.043+0000 7fbf63474640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf5c1d7880 con 0x7fbf5c0a98f0 2024-03-25T18:14:35.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.043+0000 7fbf63474640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:35.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.043+0000 7fbf63474640 10 monclient(hunting): authenticate will time out at 7230.295382s 2024-03-25T18:14:35.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.054+0000 7fbf60a0f640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fbf5c13f710 0x7fbf5c1d7da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:36468/0 (socket says 172.21.15.37:36468) 2024-03-25T18:14:35.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.054+0000 7fbf60a0f640 1 -- 172.21.15.37:0/527001638 learned_addr learned my addr 172.21.15.37:0/527001638 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:35.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.055+0000 7fbf51ffb640 10 client.?.objecter ms_handle_connect 0x7fbf5c13f710 2024-03-25T18:14:35.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.055+0000 7fbf51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:35.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.068+0000 7fbf51ffb640 10 client.?.objecter ms_handle_connect 0x7fbf5c1430c0 2024-03-25T18:14:35.069 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.068+0000 7fbf51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:35.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.069+0000 7fbf51ffb640 10 client.?.objecter ms_handle_connect 0x7fbf5c0a98f0 2024-03-25T18:14:35.070 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.069+0000 7fbf51ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:35.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.081+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803417536 0 0) 0x7fbf5c1d7880 con 0x7fbf5c0a98f0 2024-03-25T18:14:35.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.081+0000 7fbf51ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:35.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.081+0000 7fbf51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:35.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.081+0000 7fbf51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:35.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.081+0000 7fbf51ffb640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:14:35.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.081+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf40004eb0 con 0x7fbf5c0a98f0 2024-03-25T18:14:35.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.084+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 260947820 0 0) 0x7fbf5c0ab9e0 con 0x7fbf5c13f710 2024-03-25T18:14:35.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.084+0000 7fbf51ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:35.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.084+0000 7fbf51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:35.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.084+0000 7fbf51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:35.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.084+0000 7fbf51ffb640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:14:35.085 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.084+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c1d7880 con 0x7fbf5c13f710 2024-03-25T18:14:35.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.092+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2500755221 0 0) 0x7fbf5c138a50 con 0x7fbf5c1430c0 2024-03-25T18:14:35.093 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.092+0000 7fbf51ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:35.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.092+0000 7fbf51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:35.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.092+0000 7fbf51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:35.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.092+0000 7fbf51ffb640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:14:35.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.092+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf5c0ab9e0 con 0x7fbf5c1430c0 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.100+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1157920609 0 0) 0x7fbf5c1d7880 con 0x7fbf5c13f710 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.100+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf5c138a50 con 0x7fbf5c13f710 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.100+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbf4c0019b0 con 0x7fbf5c13f710 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.100+0000 7fbf51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.100+0000 7fbf51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.100+0000 7fbf51ffb640 10 monclient(hunting): dump: 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:35.101 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:35.102 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:35.102 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:35.102 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:35.102 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:35.102 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.100+0000 7fbf51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:35.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 395257430 0 0) 0x7fbf5c138a50 con 0x7fbf5c13f710 2024-03-25T18:14:35.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 >> v1:172.21.15.37:6790/0 conn(0x7fbf5c0a98f0 legacy=0x7fbf5c138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:35.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 >> v1:172.21.15.37:6789/0 conn(0x7fbf5c1430c0 legacy=0x7fbf5c1e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:35.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf51ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:35.111 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf51ffb640 1 monclient: found mon.b 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf51ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf5c1d84b0 con 0x7fbf5c13f710 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf51ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf51ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf63474640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf63474640 10 monclient: _renew_subs 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf63474640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf63474640 1 -- 172.21.15.37:0/527001638 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf5c1d51f0 con 0x7fbf5c13f710 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf63474640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf63474640 10 monclient: _renew_subs 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf63474640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:35.112 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf63474640 1 -- 172.21.15.37:0/527001638 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf5c138e60 con 0x7fbf5c13f710 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.110+0000 7fbf63474640 1 librados: init done 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.111+0000 7fbf63474640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.111+0000 7fbf63474640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.111+0000 7fbf63474640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:35.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.111+0000 7fbf63474640 1 -- 172.21.15.37:0/527001638 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf5c1d76e0 con 0x7fbf5c13f710 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.422+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbf4c003610 con 0x7fbf5c13f710 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.422+0000 7fbf51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.422+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fbf4c003800 con 0x7fbf5c13f710 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.422+0000 7fbf51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.422+0000 7fbf51ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.422+0000 7fbf51ffb640 10 monclient: dump: 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:35.424 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:35.425 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:35.425 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:35.425 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:35.425 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:35.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.422+0000 7fbf537fe640 4 set_mon_vals no callback set 2024-03-25T18:14:35.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.423+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fbf4c003a20 con 0x7fbf5c13f710 2024-03-25T18:14:35.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.424+0000 7fbf61210640 10 monclient: get_auth_request con 0x7fbf4006de20 auth_method 0 2024-03-25T18:14:35.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.424+0000 7fbf51ffb640 10 client.4537.objecter ms_handle_connect 0x7fbf4006de20 2024-03-25T18:14:35.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.428+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fbf4c054680 con 0x7fbf5c13f710 2024-03-25T18:14:35.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.428+0000 7fbf51ffb640 10 client.4537.objecter ms_dispatch 0x7fbf5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:35.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.428+0000 7fbf51ffb640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:35.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.428+0000 7fbf51ffb640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:35.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.428+0000 7fbf51ffb640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-25T18:14:35.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.429+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbf4c050190 con 0x7fbf5c13f710 2024-03-25T18:14:35.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.429+0000 7fbf51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:35.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.429+0000 7fbf51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:35.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.678+0000 7fbf63474640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:35.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.678+0000 7fbf63474640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:35.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.678+0000 7fbf63474640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:14:35.679 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.678+0000 7fbf63474640 1 -- 172.21.15.37:0/527001638 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf5c0627b0 con 0x7fbf5c13f710 2024-03-25T18:14:35.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.766+0000 7fbf51ffb640 1 -- 172.21.15.37:0/527001638 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fbf4c0520c0 con 0x7fbf5c13f710 2024-03-25T18:14:35.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.766+0000 7fbf51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:35.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.766+0000 7fbf51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:35.768 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:14:35.768 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:35.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.768+0000 7fbf63474640 10 librados: watch_flush enter 2024-03-25T18:14:35.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 10 librados: watch_flush exit 2024-03-25T18:14:35.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-25T18:14:35.770 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 10 client.4537.objecter successfully canceled tick 2024-03-25T18:14:35.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 1 -- 172.21.15.37:0/527001638 >> v1:172.21.15.117:6800/3703993921 conn(0x7fbf4006de20 legacy=0x7fbf400702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:35.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 10 monclient: shutdown 2024-03-25T18:14:35.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:35.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 1 -- 172.21.15.37:0/527001638 >> v1:172.21.15.117:6789/0 conn(0x7fbf5c13f710 legacy=0x7fbf5c1d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:35.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 1 -- 172.21.15.37:0/527001638 shutdown_connections 2024-03-25T18:14:35.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 1 -- 172.21.15.37:0/527001638 >> 172.21.15.37:0/527001638 conn(0x7fbf5c0b7ea0 msgr2=0x7fbf5c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:35.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 1 -- 172.21.15.37:0/527001638 shutdown_connections 2024-03-25T18:14:35.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 1 -- 172.21.15.37:0/527001638 wait complete. 2024-03-25T18:14:35.771 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:35.769+0000 7fbf63474640 1 librados: shutdown 2024-03-25T18:14:35.783 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:42.590 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.703+0000 7f7f0cb4d640 10 monclient: get_monmap_and_config 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.703+0000 7f7f0cb4d640 10 monclient: build_initial_monmap 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.703+0000 7f7f0cb4d640 10 monclient: monmap: 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:14:42.704705+0000 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:14:42.704705+0000 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:14:42.705 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:14:42.706 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:42.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.704+0000 7f7f0cb4d640 1 Processor -- start 2024-03-25T18:14:42.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.704+0000 7f7f0cb4d640 1 -- start start 2024-03-25T18:14:42.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.704+0000 7f7f0cb4d640 10 monclient: init 2024-03-25T18:14:42.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.705+0000 7f7f0cb4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:42.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.705+0000 7f7f0cb4d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:42.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.705+0000 7f7f0cb4d640 10 monclient(hunting): picked mon.noname-a con 0x7f7f080a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:42.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.705+0000 7f7f0cb4d640 10 monclient(hunting): picked mon.noname-b con 0x7f7f0813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:42.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.705+0000 7f7f0cb4d640 10 monclient(hunting): picked mon.noname-c con 0x7f7f081430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:42.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.705+0000 7f7f0cb4d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f08142520 con 0x7f7f080a98f0 2024-03-25T18:14:42.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.705+0000 7f7f0cb4d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f080b28a0 con 0x7f7f081430c0 2024-03-25T18:14:42.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.705+0000 7f7f0cb4d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f080b2a30 con 0x7f7f0813f710 2024-03-25T18:14:42.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.705+0000 7f7f0cb4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:42.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.705+0000 7f7f0cb4d640 10 monclient(hunting): authenticate will time out at 7237.957897s 2024-03-25T18:14:42.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.716+0000 7f7f0659c640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f7f080a98f0 0x7f7f080a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:40246/0 (socket says 172.21.15.37:40246) 2024-03-25T18:14:42.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.716+0000 7f7f0659c640 1 -- 172.21.15.37:0/2636726758 learned_addr learned my addr 172.21.15.37:0/2636726758 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:42.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.730+0000 7f7f05d9b640 1 -- 172.21.15.37:0/2636726758 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 815273603 0 0) 0x7f7f080b2a30 con 0x7f7f0813f710 2024-03-25T18:14:42.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.730+0000 7f7f05d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.730+0000 7f7f05d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.730+0000 7f7f05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.730+0000 7f7f05d9b640 10 monclient(hunting): my global_id is 4535 2024-03-25T18:14:42.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.730+0000 7f7f05d9b640 1 -- 172.21.15.37:0/2636726758 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef4003520 con 0x7f7f0813f710 2024-03-25T18:14:42.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.734+0000 7f7f05d9b640 1 -- 172.21.15.37:0/2636726758 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805668161 0 0) 0x7f7f08142520 con 0x7f7f080a98f0 2024-03-25T18:14:42.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.734+0000 7f7f05d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.734+0000 7f7f05d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.734+0000 7f7f05d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.734+0000 7f7f05d9b640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:14:42.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.734+0000 7f7f05d9b640 1 -- 172.21.15.37:0/2636726758 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f080b2a30 con 0x7f7f080a98f0 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 1 -- 172.21.15.37:0/2636726758 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3241883801 0 0) 0x7f7ef4003520 con 0x7f7f0813f710 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 1 -- 172.21.15.37:0/2636726758 >> v1:172.21.15.117:6789/0 conn(0x7f7f081430c0 legacy=0x7f7f0814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 1 -- 172.21.15.37:0/2636726758 >> v1:172.21.15.37:6789/0 conn(0x7f7f080a98f0 legacy=0x7f7f080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 1 monclient: found mon.noname-b 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 1 -- 172.21.15.37:0/2636726758 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f080b44c0 con 0x7f7f0813f710 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 10 monclient: _finish_auth 0 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 1 -- 172.21.15.37:0/2636726758 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7ef0001690 con 0x7f7f0813f710 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f0cb4d640 5 monclient: authenticate success, global_id 4535 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f0cb4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:42.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f05d9b640 10 monclient: dump: 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:42.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.749+0000 7f7f0cb4d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.762+0000 7f7f05d9b640 1 -- 172.21.15.37:0/2636726758 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ef00017f0 con 0x7f7f0813f710 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.762+0000 7f7f05d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.762+0000 7f7f05d9b640 1 -- 172.21.15.37:0/2636726758 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7ef0001c00 con 0x7f7f0813f710 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.762+0000 7f7f05d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.762+0000 7f7f05d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.762+0000 7f7f05d9b640 10 monclient: dump: 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:42.763 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:42.764 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:42.764 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:42.764 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:42.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.762+0000 7f7f0cb4d640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:42.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.762+0000 7f7f0cb4d640 4 set_mon_vals no callback set 2024-03-25T18:14:42.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.762+0000 7f7f0cb4d640 10 monclient: shutdown 2024-03-25T18:14:42.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.762+0000 7f7f0cb4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:42.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.762+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2636726758 >> v1:172.21.15.37:6790/0 conn(0x7f7f0813f710 legacy=0x7f7f081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.763+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2636726758 shutdown_connections 2024-03-25T18:14:42.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.763+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2636726758 >> 172.21.15.37:0/2636726758 conn(0x7f7f080b7ea0 msgr2=0x7f7f080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:42.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.763+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2636726758 shutdown_connections 2024-03-25T18:14:42.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.763+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2636726758 wait complete. 2024-03-25T18:14:42.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.764+0000 7f7f0cb4d640 10 monclient: build_initial_monmap 2024-03-25T18:14:42.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.764+0000 7f7f0cb4d640 10 monclient: monmap: 2024-03-25T18:14:42.766 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:42.766 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:42.766 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:14:42.766 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:14:42.766 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:42.766 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:14:42.766 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:42.766 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:14:42.767 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:14:42.767 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:42.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.764+0000 7f7f0cb4d640 1 librados: starting msgr at 2024-03-25T18:14:42.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.764+0000 7f7f0cb4d640 1 librados: starting objecter 2024-03-25T18:14:42.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 1 Processor -- start 2024-03-25T18:14:42.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 1 -- start start 2024-03-25T18:14:42.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 1 librados: setting wanted keys 2024-03-25T18:14:42.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 1 librados: calling monclient init 2024-03-25T18:14:42.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 10 monclient: init 2024-03-25T18:14:42.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:42.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:42.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 10 monclient(hunting): picked mon.noname-b con 0x7f7f080a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:42.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 10 monclient(hunting): picked mon.noname-c con 0x7f7f0813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:42.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 10 monclient(hunting): picked mon.noname-a con 0x7f7f081430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:42.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f081398d0 con 0x7f7f081430c0 2024-03-25T18:14:42.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f0806ffe0 con 0x7f7f080a98f0 2024-03-25T18:14:42.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f080702c0 con 0x7f7f0813f710 2024-03-25T18:14:42.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:42.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.765+0000 7f7f0cb4d640 10 monclient(hunting): authenticate will time out at 7238.018271s 2024-03-25T18:14:42.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.775+0000 7f7efffff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f7f0813f710 0x7f7f08133f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:33238/0 (socket says 172.21.15.37:33238) 2024-03-25T18:14:42.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.775+0000 7f7efffff640 1 -- 172.21.15.37:0/2544286177 learned_addr learned my addr 172.21.15.37:0/2544286177 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:42.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.775+0000 7f7efeffd640 10 client.?.objecter ms_handle_connect 0x7f7f0813f710 2024-03-25T18:14:42.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.775+0000 7f7efeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.779+0000 7f7efeffd640 10 client.?.objecter ms_handle_connect 0x7f7f080a98f0 2024-03-25T18:14:42.781 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.779+0000 7f7efeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.782+0000 7f7efeffd640 10 client.?.objecter ms_handle_connect 0x7f7f081430c0 2024-03-25T18:14:42.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.782+0000 7f7efeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:42.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.791+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2380498620 0 0) 0x7f7f080702c0 con 0x7f7f0813f710 2024-03-25T18:14:42.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.791+0000 7f7efeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.791+0000 7f7efeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.792 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.791+0000 7f7efeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.791+0000 7f7efeffd640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:14:42.793 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.791+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee8004eb0 con 0x7f7f0813f710 2024-03-25T18:14:42.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.793+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3901325128 0 0) 0x7f7f081398d0 con 0x7f7f081430c0 2024-03-25T18:14:42.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.793+0000 7f7efeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.793+0000 7f7efeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.793+0000 7f7efeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.793+0000 7f7efeffd640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:14:42.794 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.793+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f080702c0 con 0x7f7f081430c0 2024-03-25T18:14:42.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.796+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901995437 0 0) 0x7f7f0806ffe0 con 0x7f7f080a98f0 2024-03-25T18:14:42.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.796+0000 7f7efeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:42.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.796+0000 7f7efeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:42.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.796+0000 7f7efeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:42.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.796+0000 7f7efeffd640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:14:42.798 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.796+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7f081398d0 con 0x7f7f080a98f0 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.805+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 58211112 0 0) 0x7f7ee8004eb0 con 0x7f7f0813f710 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.805+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f0806ffe0 con 0x7f7f0813f710 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.805+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7ef0001690 con 0x7f7f0813f710 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.805+0000 7f7efeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.805+0000 7f7efeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.805+0000 7f7efeffd640 10 monclient(hunting): dump: 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.807 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:42.808 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:42.808 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:42.808 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:42.808 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:42.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.805+0000 7f7efeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:42.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.813+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4275183485 0 0) 0x7f7f080702c0 con 0x7f7f081430c0 2024-03-25T18:14:42.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.813+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ee8004eb0 con 0x7f7f081430c0 2024-03-25T18:14:42.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.813+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7f00001b00 con 0x7f7f081430c0 2024-03-25T18:14:42.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.813+0000 7f7efeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.813+0000 7f7efeffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.813+0000 7f7efeffd640 10 monclient(hunting): dump: 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:42.815 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:42.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.820+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4081934691 0 0) 0x7f7f081398d0 con 0x7f7f080a98f0 2024-03-25T18:14:42.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.821+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7f080702c0 con 0x7f7f080a98f0 2024-03-25T18:14:42.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.821+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7ef8001930 con 0x7f7f080a98f0 2024-03-25T18:14:42.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.821+0000 7f7efeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.821+0000 7f7efeffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:42.822 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.821+0000 7f7efeffd640 10 monclient(hunting): dump: 2024-03-25T18:14:42.822 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:42.822 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:42.822 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.822 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.823 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.823 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:42.823 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:42.823 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:42.823 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:42.823 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 835208206 0 0) 0x7f7ee8004eb0 con 0x7f7f081430c0 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 >> v1:172.21.15.37:6790/0 conn(0x7f7f0813f710 legacy=0x7f7f08133f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 >> v1:172.21.15.117:6789/0 conn(0x7f7f080a98f0 legacy=0x7f7f0806e8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7efeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7efeffd640 1 monclient: found mon.a 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7efeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7efeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f0814d3f0 con 0x7f7f081430c0 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7efeffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7efeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7efeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7f0cb4d640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:14:42.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7f0cb4d640 10 monclient: _renew_subs 2024-03-25T18:14:42.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7f0cb4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:42.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2544286177 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f0814dbb0 con 0x7f7f081430c0 2024-03-25T18:14:42.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7f0cb4d640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:42.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7f0cb4d640 10 monclient: _renew_subs 2024-03-25T18:14:42.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7f0cb4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:42.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2544286177 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f08070930 con 0x7f7f081430c0 2024-03-25T18:14:42.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.822+0000 7f7f0cb4d640 1 librados: init done 2024-03-25T18:14:42.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.823+0000 7f7f0cb4d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.823+0000 7f7f0cb4d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:42.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.823+0000 7f7f0cb4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:42.826 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.823+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2544286177 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f0814f310 con 0x7f7f081430c0 2024-03-25T18:14:42.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.852+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7f00001fb0 con 0x7f7f081430c0 2024-03-25T18:14:42.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.852+0000 7f7efeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:42.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.852+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7f00003b10 con 0x7f7f081430c0 2024-03-25T18:14:42.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.852+0000 7f7efeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:42.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.852+0000 7f7efeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:42.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.852+0000 7f7efeffd640 10 monclient: dump: 2024-03-25T18:14:42.853 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:42.854 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:42.854 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.854 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:42.854 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:42.854 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:42.854 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:42.854 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:42.854 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:42.854 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:42.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.852+0000 7f7f0559a640 4 set_mon_vals no callback set 2024-03-25T18:14:42.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.852+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f7f00019ae0 con 0x7f7f081430c0 2024-03-25T18:14:42.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.853+0000 7f7f0659c640 10 monclient: get_auth_request con 0x7f7ee806e110 auth_method 0 2024-03-25T18:14:42.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.853+0000 7f7efeffd640 10 client.4542.objecter ms_handle_connect 0x7f7ee806e110 2024-03-25T18:14:42.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.856+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f7f00053770 con 0x7f7f081430c0 2024-03-25T18:14:42.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.856+0000 7f7efeffd640 10 client.4542.objecter ms_dispatch 0x7f7f08000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:42.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.856+0000 7f7efeffd640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:42.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.856+0000 7f7efeffd640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:42.857 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:42.856+0000 7f7efeffd640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-25T18:14:43.150 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.149+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7f0004f0b0 con 0x7f7f081430c0 2024-03-25T18:14:43.150 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.149+0000 7f7efeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:43.150 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.149+0000 7f7efeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:43.418 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.417+0000 7f7f0cb4d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:43.418 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.417+0000 7f7f0cb4d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:43.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.417+0000 7f7f0cb4d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:43.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.417+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2544286177 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f080627b0 con 0x7f7f081430c0 2024-03-25T18:14:43.456 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.455+0000 7f7efeffd640 1 -- 172.21.15.37:0/2544286177 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f7f000511b0 con 0x7f7f081430c0 2024-03-25T18:14:43.456 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.455+0000 7f7efeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:43.456 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.455+0000 7f7efeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:43.457 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:14:43.457 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:43.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.457+0000 7f7f0cb4d640 10 librados: watch_flush enter 2024-03-25T18:14:43.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.457+0000 7f7f0cb4d640 10 librados: watch_flush exit 2024-03-25T18:14:43.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.457+0000 7f7f0cb4d640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-25T18:14:43.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.457+0000 7f7f0cb4d640 10 client.4542.objecter successfully canceled tick 2024-03-25T18:14:43.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.457+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2544286177 >> v1:172.21.15.117:6800/3703993921 conn(0x7f7ee806e110 legacy=0x7f7ee80705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:43.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.457+0000 7f7f0cb4d640 10 monclient: shutdown 2024-03-25T18:14:43.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.457+0000 7f7f0cb4d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:43.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.457+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2544286177 >> v1:172.21.15.37:6789/0 conn(0x7f7f081430c0 legacy=0x7f7f081e74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:43.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.457+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2544286177 shutdown_connections 2024-03-25T18:14:43.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.457+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2544286177 >> 172.21.15.37:0/2544286177 conn(0x7f7f080b7ea0 msgr2=0x7f7f08074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:43.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.457+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2544286177 shutdown_connections 2024-03-25T18:14:43.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.458+0000 7f7f0cb4d640 1 -- 172.21.15.37:0/2544286177 wait complete. 2024-03-25T18:14:43.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:43.458+0000 7f7f0cb4d640 1 librados: shutdown 2024-03-25T18:14:43.471 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:50.278 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:50.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.392+0000 7fe557f15640 10 monclient: get_monmap_and_config 2024-03-25T18:14:50.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.392+0000 7fe557f15640 10 monclient: build_initial_monmap 2024-03-25T18:14:50.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.392+0000 7fe557f15640 10 monclient: monmap: 2024-03-25T18:14:50.393 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:50.393 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:50.393 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:14:50.392981+0000 2024-03-25T18:14:50.393 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:14:50.392981+0000 2024-03-25T18:14:50.393 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:50.393 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:50.393 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:50.393 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:14:50.394 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:14:50.394 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:50.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.392+0000 7fe557f15640 1 Processor -- start 2024-03-25T18:14:50.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 1 -- start start 2024-03-25T18:14:50.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 10 monclient: init 2024-03-25T18:14:50.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:50.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:50.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 10 monclient(hunting): picked mon.noname-b con 0x7fe5500b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:50.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 10 monclient(hunting): picked mon.noname-c con 0x7fe5500b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:50.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 10 monclient(hunting): picked mon.noname-a con 0x7fe55014be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:50.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5500af940 con 0x7fe55014be90 2024-03-25T18:14:50.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5500b0a70 con 0x7fe5500b1730 2024-03-25T18:14:50.396 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5500b0d50 con 0x7fe5500b4d60 2024-03-25T18:14:50.396 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:50.396 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.393+0000 7fe557f15640 10 monclient(hunting): authenticate will time out at 7245.646127s 2024-03-25T18:14:50.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.398+0000 7fe555cb1640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fe5500b4d60 0x7fe55013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:33886/0 (socket says 172.21.15.37:33886) 2024-03-25T18:14:50.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.398+0000 7fe555cb1640 1 -- 172.21.15.37:0/671023525 learned_addr learned my addr 172.21.15.37:0/671023525 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:50.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.433+0000 7fe554caf640 1 -- 172.21.15.37:0/671023525 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2812492094 0 0) 0x7fe5500af940 con 0x7fe55014be90 2024-03-25T18:14:50.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.433+0000 7fe554caf640 10 monclient(hunting): _negotiate 2024-03-25T18:14:50.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.433+0000 7fe554caf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:50.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.433+0000 7fe554caf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:50.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.433+0000 7fe554caf640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:14:50.434 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.433+0000 7fe554caf640 1 -- 172.21.15.37:0/671023525 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe53c003520 con 0x7fe55014be90 2024-03-25T18:14:50.445 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 1 -- 172.21.15.37:0/671023525 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 657470115 0 0) 0x7fe53c003520 con 0x7fe55014be90 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 1 -- 172.21.15.37:0/671023525 >> v1:172.21.15.37:6790/0 conn(0x7fe5500b4d60 legacy=0x7fe55013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 1 -- 172.21.15.37:0/671023525 >> v1:172.21.15.117:6789/0 conn(0x7fe5500b1730 legacy=0x7fe5500b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 10 monclient: _finish_hunting 0 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 1 monclient: found mon.noname-a 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 1 -- 172.21.15.37:0/671023525 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5500b1030 con 0x7fe55014be90 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 10 monclient: _finish_auth 0 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 10 monclient: _check_auth_tickets 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 1 -- 172.21.15.37:0/671023525 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe548001a10 con 0x7fe55014be90 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe557f15640 5 monclient: authenticate success, global_id 4545 2024-03-25T18:14:50.446 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe557f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe554caf640 10 monclient: dump: 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:50.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.444+0000 7fe557f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:50.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.456+0000 7fe554caf640 1 -- 172.21.15.37:0/671023525 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe548001b70 con 0x7fe55014be90 2024-03-25T18:14:50.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.456+0000 7fe554caf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:50.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.456+0000 7fe554caf640 1 -- 172.21.15.37:0/671023525 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe548001fc0 con 0x7fe55014be90 2024-03-25T18:14:50.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.456+0000 7fe557f15640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:50.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.456+0000 7fe557f15640 4 set_mon_vals no callback set 2024-03-25T18:14:50.458 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.456+0000 7fe554caf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.456+0000 7fe554caf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.456+0000 7fe554caf640 10 monclient: dump: 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.457+0000 7fe557f15640 10 monclient: shutdown 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.457+0000 7fe557f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.457+0000 7fe557f15640 1 -- 172.21.15.37:0/671023525 >> v1:172.21.15.37:6789/0 conn(0x7fe55014be90 legacy=0x7fe55014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.457+0000 7fe557f15640 1 -- 172.21.15.37:0/671023525 shutdown_connections 2024-03-25T18:14:50.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.457+0000 7fe557f15640 1 -- 172.21.15.37:0/671023525 >> 172.21.15.37:0/671023525 conn(0x7fe5500617a0 msgr2=0x7fe550061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:50.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.457+0000 7fe557f15640 1 -- 172.21.15.37:0/671023525 shutdown_connections 2024-03-25T18:14:50.460 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.457+0000 7fe557f15640 1 -- 172.21.15.37:0/671023525 wait complete. 2024-03-25T18:14:50.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.458+0000 7fe557f15640 10 monclient: build_initial_monmap 2024-03-25T18:14:50.461 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.458+0000 7fe557f15640 10 monclient: monmap: 2024-03-25T18:14:50.461 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.458+0000 7fe557f15640 1 librados: starting msgr at 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.458+0000 7fe557f15640 1 librados: starting objecter 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.458+0000 7fe557f15640 1 Processor -- start 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 1 -- start start 2024-03-25T18:14:50.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 1 librados: setting wanted keys 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 1 librados: calling monclient init 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 10 monclient: init 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 10 monclient(hunting): picked mon.noname-c con 0x7fe5500b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 10 monclient(hunting): picked mon.noname-b con 0x7fe5500b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 10 monclient(hunting): picked mon.noname-a con 0x7fe55014be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe550140810 con 0x7fe55014be90 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe550140380 con 0x7fe5500b4d60 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe550140510 con 0x7fe5500b1730 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.459+0000 7fe557f15640 10 monclient(hunting): authenticate will time out at 7245.712017s 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.460+0000 7fe555cb1640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fe5500b1730 0x7fe550134ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:33946/0 (socket says 172.21.15.37:33946) 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.460+0000 7fe555cb1640 1 -- 172.21.15.37:0/64521853 learned_addr learned my addr 172.21.15.37:0/64521853 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.460+0000 7fe5467fc640 10 client.?.objecter ms_handle_connect 0x7fe5500b4d60 2024-03-25T18:14:50.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.460+0000 7fe5467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:50.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.460+0000 7fe5467fc640 10 client.?.objecter ms_handle_connect 0x7fe5500b1730 2024-03-25T18:14:50.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.460+0000 7fe5467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:50.505 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:07:25.850 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:50.512 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:07:25.857 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:50.532 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.531+0000 7fe5467fc640 10 client.?.objecter ms_handle_connect 0x7fe55014be90 2024-03-25T18:14:50.533 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.531+0000 7fe5467fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:50.545 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:07:25.890 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:50.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.562+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2974199590 0 0) 0x7fe550140810 con 0x7fe55014be90 2024-03-25T18:14:50.563 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.562+0000 7fe5467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:50.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.562+0000 7fe5467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:50.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.562+0000 7fe5467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:50.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.562+0000 7fe5467fc640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:14:50.564 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.562+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe534004eb0 con 0x7fe55014be90 2024-03-25T18:14:50.572 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:07:25.885 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:50.574 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:07:25.919 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:50.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.585+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3491212455 0 0) 0x7fe550140380 con 0x7fe5500b4d60 2024-03-25T18:14:50.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.585+0000 7fe5467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:50.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.585+0000 7fe5467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:50.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.585+0000 7fe5467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:50.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.585+0000 7fe5467fc640 10 monclient(hunting): my global_id is 4546 2024-03-25T18:14:50.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.585+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe550140810 con 0x7fe5500b4d60 2024-03-25T18:14:50.589 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:07:25.902 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:50.590 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:07:25.935 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:50.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.601+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 443017542 0 0) 0x7fe550140510 con 0x7fe5500b1730 2024-03-25T18:14:50.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.601+0000 7fe5467fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:50.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.601+0000 7fe5467fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:50.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.601+0000 7fe5467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:50.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.601+0000 7fe5467fc640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:14:50.603 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.601+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe550140380 con 0x7fe5500b1730 2024-03-25T18:14:50.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.606+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3201013385 0 0) 0x7fe550140810 con 0x7fe5500b4d60 2024-03-25T18:14:50.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.606+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe550140510 con 0x7fe5500b4d60 2024-03-25T18:14:50.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.606+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe5400018a0 con 0x7fe5500b4d60 2024-03-25T18:14:50.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.606+0000 7fe5467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:50.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.606+0000 7fe5467fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:50.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.606+0000 7fe5467fc640 10 monclient(hunting): dump: 2024-03-25T18:14:50.607 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:50.607 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:50.607 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:50.607 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:50.608 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:50.608 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:50.608 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:50.608 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:50.608 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:50.608 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:50.608 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.606+0000 7fe5467fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:50.619 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:07:25.932 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:50.620 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:07:25.965 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.625+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3851474635 0 0) 0x7fe550140380 con 0x7fe5500b1730 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.625+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe550140810 con 0x7fe5500b1730 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.625+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe54c0018b0 con 0x7fe5500b1730 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.625+0000 7fe5467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.625+0000 7fe5467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.625+0000 7fe5467fc640 10 monclient(hunting): dump: 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:50.627 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:50.628 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:50.628 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:50.628 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:50.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2070786884 0 0) 0x7fe550140810 con 0x7fe5500b1730 2024-03-25T18:14:50.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 >> v1:172.21.15.117:6789/0 conn(0x7fe5500b4d60 legacy=0x7fe5501d7980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 >> v1:172.21.15.37:6789/0 conn(0x7fe55014be90 legacy=0x7fe5501e31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:50.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe5467fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:50.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe5467fc640 1 monclient: found mon.c 2024-03-25T18:14:50.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe5467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe5467fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5501355e0 con 0x7fe5500b1730 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe5467fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe5467fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe5467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe557f15640 5 monclient: authenticate success, global_id 4541 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe557f15640 10 monclient: _renew_subs 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe557f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.644+0000 7fe557f15640 1 -- 172.21.15.37:0/64521853 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5501357d0 con 0x7fe5500b1730 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.645+0000 7fe557f15640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.645+0000 7fe557f15640 10 monclient: _renew_subs 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.645+0000 7fe557f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.645+0000 7fe557f15640 1 -- 172.21.15.37:0/64521853 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe550144da0 con 0x7fe5500b1730 2024-03-25T18:14:50.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.645+0000 7fe557f15640 1 librados: init done 2024-03-25T18:14:50.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.645+0000 7fe557f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:50.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.645+0000 7fe557f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:50.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.645+0000 7fe557f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:14:50.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:50.645+0000 7fe557f15640 1 -- 172.21.15.37:0/64521853 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe550141c10 con 0x7fe5500b1730 2024-03-25T18:14:51.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.070+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe54c0047e0 con 0x7fe5500b1730 2024-03-25T18:14:51.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.070+0000 7fe5467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:51.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.070+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe54c004af0 con 0x7fe5500b1730 2024-03-25T18:14:51.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.070+0000 7fe5467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:51.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.070+0000 7fe5467fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:51.071 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.070+0000 7fe5467fc640 10 monclient: dump: 2024-03-25T18:14:51.071 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:51.072 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:51.072 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:51.072 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:51.072 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:51.072 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:51.072 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:51.072 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:51.072 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:51.072 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:51.072 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.070+0000 7fe547fff640 4 set_mon_vals no callback set 2024-03-25T18:14:51.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.070+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fe54c01aac0 con 0x7fe5500b1730 2024-03-25T18:14:51.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.071+0000 7fe5554b0640 10 monclient: get_auth_request con 0x7fe53406e060 auth_method 0 2024-03-25T18:14:51.073 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.071+0000 7fe5467fc640 10 client.4541.objecter ms_handle_connect 0x7fe53406e060 2024-03-25T18:14:51.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.078+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fe54c0547b0 con 0x7fe5500b1730 2024-03-25T18:14:51.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.078+0000 7fe5467fc640 10 client.4541.objecter ms_dispatch 0x7fe550000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:51.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.078+0000 7fe5467fc640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:51.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.078+0000 7fe5467fc640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:51.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.078+0000 7fe5467fc640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-25T18:14:51.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.078+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe54c0500f0 con 0x7fe5500b1730 2024-03-25T18:14:51.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.078+0000 7fe5467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:51.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.078+0000 7fe5467fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:51.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.315+0000 7fe557f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:51.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.315+0000 7fe557f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:51.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.315+0000 7fe557f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:14:51.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.315+0000 7fe557f15640 1 -- 172.21.15.37:0/64521853 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5500627b0 con 0x7fe5500b1730 2024-03-25T18:14:51.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.371+0000 7fe5467fc640 1 -- 172.21.15.37:0/64521853 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fe54c0521f0 con 0x7fe5500b1730 2024-03-25T18:14:51.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.371+0000 7fe5467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:51.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.371+0000 7fe5467fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:51.372 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:14:51.372 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:51.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 10 librados: watch_flush enter 2024-03-25T18:14:51.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 10 librados: watch_flush exit 2024-03-25T18:14:51.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-25T18:14:51.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 10 client.4541.objecter successfully canceled tick 2024-03-25T18:14:51.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 1 -- 172.21.15.37:0/64521853 >> v1:172.21.15.117:6800/3703993921 conn(0x7fe53406e060 legacy=0x7fe534070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:51.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 10 monclient: shutdown 2024-03-25T18:14:51.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:51.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 1 -- 172.21.15.37:0/64521853 >> v1:172.21.15.37:6790/0 conn(0x7fe5500b1730 legacy=0x7fe550134ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:51.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 1 -- 172.21.15.37:0/64521853 shutdown_connections 2024-03-25T18:14:51.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 1 -- 172.21.15.37:0/64521853 >> 172.21.15.37:0/64521853 conn(0x7fe5500617a0 msgr2=0x7fe5500b1bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:51.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 1 -- 172.21.15.37:0/64521853 shutdown_connections 2024-03-25T18:14:51.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 1 -- 172.21.15.37:0/64521853 wait complete. 2024-03-25T18:14:51.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:51.373+0000 7fe557f15640 1 librados: shutdown 2024-03-25T18:14:51.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:58.192 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:58.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.306+0000 7f2136faf640 10 monclient: get_monmap_and_config 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.306+0000 7f2136faf640 10 monclient: build_initial_monmap 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.306+0000 7f2136faf640 10 monclient: monmap: 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:14:58.307450+0000 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:14:58.307450+0000 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:14:58.308 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.307+0000 7f2136faf640 1 Processor -- start 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.307+0000 7f2136faf640 1 -- start start 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.307+0000 7f2136faf640 10 monclient: init 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.308+0000 7f2136faf640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.308+0000 7f2136faf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.308+0000 7f2136faf640 10 monclient(hunting): picked mon.noname-a con 0x7f21300a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.308+0000 7f2136faf640 10 monclient(hunting): picked mon.noname-b con 0x7f213013f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.308+0000 7f2136faf640 10 monclient(hunting): picked mon.noname-c con 0x7f21301430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.308+0000 7f2136faf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2130142520 con 0x7f21300a98f0 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.308+0000 7f2136faf640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21300b28a0 con 0x7f21301430c0 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.308+0000 7f2136faf640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21300b2a30 con 0x7f213013f710 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.308+0000 7f2136faf640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:58.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.308+0000 7f2136faf640 10 monclient(hunting): authenticate will time out at 7253.560802s 2024-03-25T18:14:58.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.314+0000 7f213554c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f21301430c0 0x7f213014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:50540/0 (socket says 172.21.15.37:50540) 2024-03-25T18:14:58.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.314+0000 7f213554c640 1 -- 172.21.15.37:0/3722585709 learned_addr learned my addr 172.21.15.37:0/3722585709 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:58.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.323+0000 7f212f7fe640 1 -- 172.21.15.37:0/3722585709 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101728795 0 0) 0x7f21300b28a0 con 0x7f21301430c0 2024-03-25T18:14:58.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.323+0000 7f212f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:58.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.323+0000 7f212f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:58.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.323+0000 7f212f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:58.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.323+0000 7f212f7fe640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:14:58.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.323+0000 7f212f7fe640 1 -- 172.21.15.37:0/3722585709 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2120003520 con 0x7f21301430c0 2024-03-25T18:14:58.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 1 -- 172.21.15.37:0/3722585709 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2422425528 0 0) 0x7f2120003520 con 0x7f21301430c0 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 1 -- 172.21.15.37:0/3722585709 >> v1:172.21.15.37:6790/0 conn(0x7f213013f710 legacy=0x7f21301429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 1 -- 172.21.15.37:0/3722585709 >> v1:172.21.15.37:6789/0 conn(0x7f21300a98f0 legacy=0x7f21300a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 1 monclient: found mon.noname-c 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 1 -- 172.21.15.37:0/3722585709 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21300b44c0 con 0x7f21301430c0 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 1 -- 172.21.15.37:0/3722585709 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f2128001a10 con 0x7f21301430c0 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f212f7fe640 10 monclient: dump: 2024-03-25T18:14:58.339 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:58.340 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:58.340 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:58.340 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:58.340 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:58.340 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:58.340 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:58.340 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:58.340 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:58.340 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:58.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f2136faf640 5 monclient: authenticate success, global_id 4549 2024-03-25T18:14:58.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.337+0000 7f2136faf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:58.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.341+0000 7f212f7fe640 1 -- 172.21.15.37:0/3722585709 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2128001b70 con 0x7f21301430c0 2024-03-25T18:14:58.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.341+0000 7f212f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:58.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.341+0000 7f2136faf640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:58.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.341+0000 7f2136faf640 4 set_mon_vals no callback set 2024-03-25T18:14:58.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.342+0000 7f2136faf640 10 monclient: shutdown 2024-03-25T18:14:58.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.342+0000 7f2136faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:58.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.342+0000 7f2136faf640 1 -- 172.21.15.37:0/3722585709 >> v1:172.21.15.117:6789/0 conn(0x7f21301430c0 legacy=0x7f213014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:58.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.342+0000 7f2136faf640 1 -- 172.21.15.37:0/3722585709 shutdown_connections 2024-03-25T18:14:58.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.342+0000 7f2136faf640 1 -- 172.21.15.37:0/3722585709 >> 172.21.15.37:0/3722585709 conn(0x7f21300b7ea0 msgr2=0x7f21300b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:58.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.342+0000 7f2136faf640 1 -- 172.21.15.37:0/3722585709 shutdown_connections 2024-03-25T18:14:58.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.342+0000 7f2136faf640 1 -- 172.21.15.37:0/3722585709 wait complete. 2024-03-25T18:14:58.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.343+0000 7f2136faf640 10 monclient: build_initial_monmap 2024-03-25T18:14:58.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.343+0000 7f2136faf640 10 monclient: monmap: 2024-03-25T18:14:58.344 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:14:58.345 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:58.345 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:14:58.345 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:14:58.345 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:58.345 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:14:58.345 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:14:58.345 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:14:58.345 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:14:58.345 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:58.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.344+0000 7f2136faf640 1 librados: starting msgr at 2024-03-25T18:14:58.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.344+0000 7f2136faf640 1 librados: starting objecter 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.344+0000 7f2136faf640 1 Processor -- start 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.344+0000 7f2136faf640 1 -- start start 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.344+0000 7f2136faf640 1 librados: setting wanted keys 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.344+0000 7f2136faf640 1 librados: calling monclient init 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.344+0000 7f2136faf640 10 monclient: init 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.345+0000 7f2136faf640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.345+0000 7f2136faf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.345+0000 7f2136faf640 10 monclient(hunting): picked mon.noname-a con 0x7f21300a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.345+0000 7f2136faf640 10 monclient(hunting): picked mon.noname-b con 0x7f213013f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.345+0000 7f2136faf640 10 monclient(hunting): picked mon.noname-c con 0x7f21301430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.345+0000 7f2136faf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21300abca0 con 0x7f21300a98f0 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.345+0000 7f2136faf640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2130138bc0 con 0x7f213013f710 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.345+0000 7f2136faf640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21301db250 con 0x7f21301430c0 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.345+0000 7f2136faf640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:58.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.345+0000 7f2136faf640 10 monclient(hunting): authenticate will time out at 7253.597488s 2024-03-25T18:14:58.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.358+0000 7f2134d4b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f21300a98f0 0x7f21301383d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:52086/0 (socket says 172.21.15.37:52086) 2024-03-25T18:14:58.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.359+0000 7f2134d4b640 1 -- 172.21.15.37:0/3299309181 learned_addr learned my addr 172.21.15.37:0/3299309181 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:14:58.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.362+0000 7f212d7fa640 10 client.?.objecter ms_handle_connect 0x7f21300a98f0 2024-03-25T18:14:58.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.362+0000 7f212d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:58.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.366+0000 7f212d7fa640 10 client.?.objecter ms_handle_connect 0x7f21301430c0 2024-03-25T18:14:58.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.366+0000 7f212d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:58.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.366+0000 7f212d7fa640 10 client.?.objecter ms_handle_connect 0x7f213013f710 2024-03-25T18:14:58.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.366+0000 7f212d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:58.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.370+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3836549993 0 0) 0x7f21300abca0 con 0x7f21300a98f0 2024-03-25T18:14:58.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.370+0000 7f212d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:58.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.370+0000 7f212d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:58.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.370+0000 7f212d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:58.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.370+0000 7f212d7fa640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:14:58.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.370+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2114004eb0 con 0x7f21300a98f0 2024-03-25T18:14:58.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.372+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2035073862 0 0) 0x7f21301db250 con 0x7f21301430c0 2024-03-25T18:14:58.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.372+0000 7f212d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:58.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.372+0000 7f212d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:58.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.372+0000 7f212d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:58.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.372+0000 7f212d7fa640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:14:58.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.372+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21300abca0 con 0x7f21301430c0 2024-03-25T18:14:58.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.381+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2682283207 0 0) 0x7f2114004eb0 con 0x7f21300a98f0 2024-03-25T18:14:58.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.381+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21301db250 con 0x7f21300a98f0 2024-03-25T18:14:58.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.381+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f2124001a30 con 0x7f21300a98f0 2024-03-25T18:14:58.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.381+0000 7f212d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:58.382 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.381+0000 7f212d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.381+0000 7f212d7fa640 10 monclient(hunting): dump: 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:58.383 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.381+0000 7f212d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:58.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.385+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3470730148 0 0) 0x7f21300abca0 con 0x7f21301430c0 2024-03-25T18:14:58.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.385+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2114004eb0 con 0x7f21301430c0 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.385+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f2128001e80 con 0x7f21301430c0 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.385+0000 7f212d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.385+0000 7f212d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.385+0000 7f212d7fa640 10 monclient(hunting): dump: 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:58.387 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 89831056 0 0) 0x7f21301db250 con 0x7f21300a98f0 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 >> v1:172.21.15.37:6790/0 conn(0x7f21301430c0 legacy=0x7f21301dab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 >> v1:172.21.15.117:6789/0 conn(0x7f213013f710 legacy=0x7f2130137ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f212d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f212d7fa640 1 monclient: found mon.a 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f212d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f212d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21301db3e0 con 0x7f21300a98f0 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f212d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:14:58.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f212d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f212d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f2136faf640 5 monclient: authenticate success, global_id 4554 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f2136faf640 10 monclient: _renew_subs 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f2136faf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f2136faf640 1 -- 172.21.15.37:0/3299309181 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21301db630 con 0x7f21300a98f0 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f2136faf640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f2136faf640 10 monclient: _renew_subs 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f2136faf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f2136faf640 1 -- 172.21.15.37:0/3299309181 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f21301e3120 con 0x7f21300a98f0 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.386+0000 7f2136faf640 1 librados: init done 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.387+0000 7f2136faf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.387+0000 7f2136faf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.387+0000 7f2136faf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:58.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.387+0000 7f2136faf640 1 -- 172.21.15.37:0/3299309181 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21300627b0 con 0x7f21300a98f0 2024-03-25T18:14:58.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.721+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2124001c70 con 0x7f21300a98f0 2024-03-25T18:14:58.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.721+0000 7f212d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:58.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.721+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f2124003a60 con 0x7f21300a98f0 2024-03-25T18:14:58.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.721+0000 7f212d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:58.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.721+0000 7f212d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:58.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.721+0000 7f212d7fa640 10 monclient: dump: 2024-03-25T18:14:58.724 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:14:58.725 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:14:58.725 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:58.725 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:14:58.725 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:58.725 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:14:58.726 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:14:58.726 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:14:58.726 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:14:58.726 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:14:58.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.721+0000 7f212effd640 4 set_mon_vals no callback set 2024-03-25T18:14:58.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.722+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f2124003c80 con 0x7f21300a98f0 2024-03-25T18:14:58.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.723+0000 7f212ffff640 10 monclient: get_auth_request con 0x7f211406df20 auth_method 0 2024-03-25T18:14:58.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.723+0000 7f212d7fa640 10 client.4554.objecter ms_handle_connect 0x7f211406df20 2024-03-25T18:14:58.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.728+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f2124054850 con 0x7f21300a98f0 2024-03-25T18:14:58.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.728+0000 7f212d7fa640 10 client.4554.objecter ms_dispatch 0x7f2130000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:58.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.728+0000 7f212d7fa640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:58.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.728+0000 7f212d7fa640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:58.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.728+0000 7f212d7fa640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-25T18:14:58.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.728+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2124050360 con 0x7f21300a98f0 2024-03-25T18:14:58.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.728+0000 7f212d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:58.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.728+0000 7f212d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:58.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.981+0000 7f2136faf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:58.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.981+0000 7f2136faf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:58.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.981+0000 7f2136faf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:14:58.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:58.981+0000 7f2136faf640 1 -- 172.21.15.37:0/3299309181 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2130074cc0 con 0x7f21300a98f0 2024-03-25T18:14:59.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.016+0000 7f212d7fa640 1 -- 172.21.15.37:0/3299309181 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f2124052290 con 0x7f21300a98f0 2024-03-25T18:14:59.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.016+0000 7f212d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:59.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.016+0000 7f212d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:59.018 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:14:59.018 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:59.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.018+0000 7f2136faf640 10 librados: watch_flush enter 2024-03-25T18:14:59.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.018+0000 7f2136faf640 10 librados: watch_flush exit 2024-03-25T18:14:59.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.018+0000 7f2136faf640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-25T18:14:59.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.018+0000 7f2136faf640 10 client.4554.objecter successfully canceled tick 2024-03-25T18:14:59.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.018+0000 7f2136faf640 1 -- 172.21.15.37:0/3299309181 >> v1:172.21.15.117:6800/3703993921 conn(0x7f211406df20 legacy=0x7f21140703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:59.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.018+0000 7f2136faf640 10 monclient: shutdown 2024-03-25T18:14:59.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.018+0000 7f2136faf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:59.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.018+0000 7f2136faf640 1 -- 172.21.15.37:0/3299309181 >> v1:172.21.15.37:6789/0 conn(0x7f21300a98f0 legacy=0x7f21301383d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:59.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.019+0000 7f2136faf640 1 -- 172.21.15.37:0/3299309181 shutdown_connections 2024-03-25T18:14:59.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.019+0000 7f2136faf640 1 -- 172.21.15.37:0/3299309181 >> 172.21.15.37:0/3299309181 conn(0x7f21300b7ea0 msgr2=0x7f2130074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:59.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.019+0000 7f2136faf640 1 -- 172.21.15.37:0/3299309181 shutdown_connections 2024-03-25T18:14:59.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.019+0000 7f2136faf640 1 -- 172.21.15.37:0/3299309181 wait complete. 2024-03-25T18:14:59.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:14:59.019+0000 7f2136faf640 1 librados: shutdown 2024-03-25T18:14:59.032 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:05.838 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:05.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.955+0000 7f1bfb19a640 10 monclient: get_monmap_and_config 2024-03-25T18:15:05.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.955+0000 7f1bfb19a640 10 monclient: build_initial_monmap 2024-03-25T18:15:05.957 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.955+0000 7f1bfb19a640 10 monclient: monmap: 2024-03-25T18:15:05.957 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:05.957 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:05.957 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:15:05.956279+0000 2024-03-25T18:15:05.958 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:15:05.956279+0000 2024-03-25T18:15:05.958 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:05.958 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:05.958 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:05.958 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:15:05.958 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:15:05.958 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:05.958 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.956+0000 7f1bfb19a640 1 Processor -- start 2024-03-25T18:15:05.958 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.956+0000 7f1bfb19a640 1 -- start start 2024-03-25T18:15:05.958 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.956+0000 7f1bfb19a640 10 monclient: init 2024-03-25T18:15:05.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.957+0000 7f1bfb19a640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:05.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.957+0000 7f1bfb19a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:05.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.957+0000 7f1bfb19a640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf40b3b20 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:05.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.957+0000 7f1bfb19a640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf406f090 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:05.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.957+0000 7f1bfb19a640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf40b3f00 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:05.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.957+0000 7f1bfb19a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf4070490 con 0x7f1bf40b3b20 2024-03-25T18:15:05.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.957+0000 7f1bfb19a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf406f520 con 0x7f1bf40b3f00 2024-03-25T18:15:05.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.957+0000 7f1bfb19a640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf40b56d0 con 0x7f1bf406f090 2024-03-25T18:15:05.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.957+0000 7f1bfb19a640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:05.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.957+0000 7f1bfb19a640 10 monclient(hunting): authenticate will time out at 7261.209639s 2024-03-25T18:15:05.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.962+0000 7f1bf3fff640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f1bf406f090 0x7f1bf4146960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:39976/0 (socket says 172.21.15.37:39976) 2024-03-25T18:15:05.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.962+0000 7f1bf3fff640 1 -- 172.21.15.37:0/1534441611 learned_addr learned my addr 172.21.15.37:0/1534441611 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:05.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1022348152 0 0) 0x7f1bf4070490 con 0x7f1bf40b3b20 2024-03-25T18:15:05.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:05.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:05.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:05.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:15:05.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1be4003520 con 0x7f1bf40b3b20 2024-03-25T18:15:05.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 814429568 0 0) 0x7f1bf40b56d0 con 0x7f1bf406f090 2024-03-25T18:15:05.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:05.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:05.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:05.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:15:05.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.980+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf4070490 con 0x7f1bf406f090 2024-03-25T18:15:05.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.989+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393463337 0 0) 0x7f1bf406f520 con 0x7f1bf40b3f00 2024-03-25T18:15:05.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.989+0000 7f1bf37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:05.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.989+0000 7f1bf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:05.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.989+0000 7f1bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:05.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.989+0000 7f1bf37fe640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:15:05.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:05.989+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf40b56d0 con 0x7f1bf40b3f00 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1649728767 0 0) 0x7f1be4003520 con 0x7f1bf40b3b20 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 >> v1:172.21.15.37:6790/0 conn(0x7f1bf406f090 legacy=0x7f1bf4146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 >> v1:172.21.15.117:6789/0 conn(0x7f1bf40b3f00 legacy=0x7f1bf4147070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 1 monclient: found mon.noname-a 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf40b5860 con 0x7f1bf40b3b20 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:06.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1be8001690 con 0x7f1bf40b3b20 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bfb19a640 5 monclient: authenticate success, global_id 4557 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bfb19a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bf37fe640 10 monclient: dump: 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:06.005 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:06.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.002+0000 7f1bfb19a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:06.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.011+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1be80017f0 con 0x7f1bf40b3b20 2024-03-25T18:15:06.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.011+0000 7f1bf37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:06.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.011+0000 7f1bf37fe640 1 -- 172.21.15.37:0/1534441611 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1be8001c00 con 0x7f1bf40b3b20 2024-03-25T18:15:06.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.011+0000 7f1bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.011+0000 7f1bf37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.011+0000 7f1bf37fe640 10 monclient: dump: 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:06.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.011+0000 7f1bfb19a640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:06.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.011+0000 7f1bfb19a640 4 set_mon_vals no callback set 2024-03-25T18:15:06.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.012+0000 7f1bfb19a640 10 monclient: shutdown 2024-03-25T18:15:06.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.012+0000 7f1bfb19a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.012+0000 7f1bfb19a640 1 -- 172.21.15.37:0/1534441611 >> v1:172.21.15.37:6789/0 conn(0x7f1bf40b3b20 legacy=0x7f1bf4143990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.012+0000 7f1bfb19a640 1 -- 172.21.15.37:0/1534441611 shutdown_connections 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.012+0000 7f1bfb19a640 1 -- 172.21.15.37:0/1534441611 >> 172.21.15.37:0/1534441611 conn(0x7f1bf40b49a0 msgr2=0x7f1bf40b4db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.012+0000 7f1bfb19a640 1 -- 172.21.15.37:0/1534441611 shutdown_connections 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.012+0000 7f1bfb19a640 1 -- 172.21.15.37:0/1534441611 wait complete. 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.013+0000 7f1bfb19a640 10 monclient: build_initial_monmap 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.013+0000 7f1bfb19a640 10 monclient: monmap: 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:06.015 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:15:06.016 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:15:06.016 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.014+0000 7f1bfb19a640 1 librados: starting msgr at 2024-03-25T18:15:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.014+0000 7f1bfb19a640 1 librados: starting objecter 2024-03-25T18:15:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.014+0000 7f1bfb19a640 1 Processor -- start 2024-03-25T18:15:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.014+0000 7f1bfb19a640 1 -- start start 2024-03-25T18:15:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.014+0000 7f1bfb19a640 1 librados: setting wanted keys 2024-03-25T18:15:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.014+0000 7f1bfb19a640 1 librados: calling monclient init 2024-03-25T18:15:06.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.014+0000 7f1bfb19a640 10 monclient: init 2024-03-25T18:15:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.015+0000 7f1bfb19a640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.015+0000 7f1bfb19a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.015+0000 7f1bfb19a640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf40b3b20 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.015+0000 7f1bfb19a640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf40b3f00 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.015+0000 7f1bfb19a640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf4147a80 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.015+0000 7f1bfb19a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf41496c0 con 0x7f1bf40b3f00 2024-03-25T18:15:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.015+0000 7f1bfb19a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf4136c90 con 0x7f1bf4147a80 2024-03-25T18:15:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.015+0000 7f1bfb19a640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf4136f00 con 0x7f1bf40b3b20 2024-03-25T18:15:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.015+0000 7f1bfb19a640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:06.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.015+0000 7f1bfb19a640 10 monclient(hunting): authenticate will time out at 7261.267955s 2024-03-25T18:15:06.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.021+0000 7f1bf8f36640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f1bf40b3b20 0x7f1bf4148e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:40014/0 (socket says 172.21.15.37:40014) 2024-03-25T18:15:06.022 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.021+0000 7f1bf8f36640 1 -- 172.21.15.37:0/3344624816 learned_addr learned my addr 172.21.15.37:0/3344624816 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:06.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.024+0000 7f1bf17fa640 10 client.?.objecter ms_handle_connect 0x7f1bf40b3b20 2024-03-25T18:15:06.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.024+0000 7f1bf17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:06.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.032+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3662101872 0 0) 0x7f1bf4136f00 con 0x7f1bf40b3b20 2024-03-25T18:15:06.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.032+0000 7f1bf17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:06.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.032+0000 7f1bf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:06.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.032+0000 7f1bf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:06.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.032+0000 7f1bf17fa640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:15:06.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.032+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd8004eb0 con 0x7f1bf40b3b20 2024-03-25T18:15:06.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.033+0000 7f1bf17fa640 10 client.?.objecter ms_handle_connect 0x7f1bf4147a80 2024-03-25T18:15:06.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.033+0000 7f1bf17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:06.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.038+0000 7f1bf17fa640 10 client.?.objecter ms_handle_connect 0x7f1bf40b3f00 2024-03-25T18:15:06.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.038+0000 7f1bf17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:06.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.043+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1988308057 0 0) 0x7f1bd8004eb0 con 0x7f1bf40b3b20 2024-03-25T18:15:06.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.043+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1bf4136f00 con 0x7f1bf40b3b20 2024-03-25T18:15:06.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.043+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1be8001bc0 con 0x7f1bf40b3b20 2024-03-25T18:15:06.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.043+0000 7f1bf17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:06.044 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.043+0000 7f1bf17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.043+0000 7f1bf17fa640 10 monclient(hunting): dump: 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:06.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.043+0000 7f1bf17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.050+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3679182061 0 0) 0x7f1bf41496c0 con 0x7f1bf40b3f00 2024-03-25T18:15:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.050+0000 7f1bf17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.050+0000 7f1bf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.050+0000 7f1bf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.050+0000 7f1bf17fa640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:15:06.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.050+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bd8004eb0 con 0x7f1bf40b3f00 2024-03-25T18:15:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.052+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181641359 0 0) 0x7f1bf4136c90 con 0x7f1bf4147a80 2024-03-25T18:15:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.052+0000 7f1bf17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.052+0000 7f1bf17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.052+0000 7f1bf17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.052+0000 7f1bf17fa640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:15:06.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.052+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf41496c0 con 0x7f1bf4147a80 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2436238073 0 0) 0x7f1bf4136f00 con 0x7f1bf40b3b20 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 >> v1:172.21.15.117:6789/0 conn(0x7f1bf4147a80 legacy=0x7f1bf41e6bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 >> v1:172.21.15.37:6789/0 conn(0x7f1bf40b3f00 legacy=0x7f1bf4149c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bf17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bf17fa640 1 monclient: found mon.c 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bf17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bf17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf414a310 con 0x7f1bf40b3b20 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bf17fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bf17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bf17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bfb19a640 5 monclient: authenticate success, global_id 4553 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bfb19a640 10 monclient: _renew_subs 2024-03-25T18:15:06.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bfb19a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:06.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bfb19a640 1 -- 172.21.15.37:0/3344624816 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf41458e0 con 0x7f1bf40b3b20 2024-03-25T18:15:06.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bfb19a640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:06.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bfb19a640 10 monclient: _renew_subs 2024-03-25T18:15:06.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bfb19a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:06.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bfb19a640 1 -- 172.21.15.37:0/3344624816 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf4145c80 con 0x7f1bf40b3b20 2024-03-25T18:15:06.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.053+0000 7f1bfb19a640 1 librados: init done 2024-03-25T18:15:06.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.054+0000 7f1bfb19a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:06.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.054+0000 7f1bfb19a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:06.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.054+0000 7f1bfb19a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:06.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.054+0000 7f1bfb19a640 1 -- 172.21.15.37:0/3344624816 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf40627b0 con 0x7f1bf40b3b20 2024-03-25T18:15:06.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.383+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1be80011f0 con 0x7f1bf40b3b20 2024-03-25T18:15:06.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.383+0000 7f1bf17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:06.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.383+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f1be8003b00 con 0x7f1bf40b3b20 2024-03-25T18:15:06.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.383+0000 7f1bf17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:06.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.383+0000 7f1bf17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:06.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.383+0000 7f1bf17fa640 10 monclient: dump: 2024-03-25T18:15:06.385 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:06.385 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:06.385 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:06.386 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:06.386 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:06.386 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:06.386 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:06.386 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:06.386 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:06.386 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:06.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.383+0000 7f1bf2ffd640 4 set_mon_vals no callback set 2024-03-25T18:15:06.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.384+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f1be8003d20 con 0x7f1bf40b3b20 2024-03-25T18:15:06.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.385+0000 7f1bf3fff640 10 monclient: get_auth_request con 0x7f1bd806de10 auth_method 0 2024-03-25T18:15:06.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.385+0000 7f1bf17fa640 10 client.4553.objecter ms_handle_connect 0x7f1bd806de10 2024-03-25T18:15:06.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.390+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f1be8054980 con 0x7f1bf40b3b20 2024-03-25T18:15:06.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.390+0000 7f1bf17fa640 10 client.4553.objecter ms_dispatch 0x7f1bf4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:06.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.390+0000 7f1bf17fa640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:06.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.390+0000 7f1bf17fa640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:06.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.390+0000 7f1bf17fa640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-25T18:15:06.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.390+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1be80502c0 con 0x7f1bf40b3b20 2024-03-25T18:15:06.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.390+0000 7f1bf17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:06.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.390+0000 7f1bf17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:06.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.608+0000 7f1bfb19a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:06.609 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.608+0000 7f1bfb19a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:06.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.608+0000 7f1bfb19a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:06.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.608+0000 7f1bfb19a640 1 -- 172.21.15.37:0/3344624816 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1bf414a4a0 con 0x7f1bf40b3b20 2024-03-25T18:15:06.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.634+0000 7f1bf17fa640 1 -- 172.21.15.37:0/3344624816 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f1be80523c0 con 0x7f1bf40b3b20 2024-03-25T18:15:06.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.634+0000 7f1bf17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:06.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.634+0000 7f1bf17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:06.636 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:15:06.636 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:06.638 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.636+0000 7f1bfb19a640 10 librados: watch_flush enter 2024-03-25T18:15:06.638 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.636+0000 7f1bfb19a640 10 librados: watch_flush exit 2024-03-25T18:15:06.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.636+0000 7f1bfb19a640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-25T18:15:06.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.636+0000 7f1bfb19a640 10 client.4553.objecter successfully canceled tick 2024-03-25T18:15:06.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.637+0000 7f1bfb19a640 1 -- 172.21.15.37:0/3344624816 >> v1:172.21.15.117:6800/3703993921 conn(0x7f1bd806de10 legacy=0x7f1bd80702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:06.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.637+0000 7f1bfb19a640 10 monclient: shutdown 2024-03-25T18:15:06.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.637+0000 7f1bfb19a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:06.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.637+0000 7f1bfb19a640 1 -- 172.21.15.37:0/3344624816 >> v1:172.21.15.37:6790/0 conn(0x7f1bf40b3b20 legacy=0x7f1bf4148e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:06.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.637+0000 7f1bfb19a640 1 -- 172.21.15.37:0/3344624816 shutdown_connections 2024-03-25T18:15:06.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.637+0000 7f1bfb19a640 1 -- 172.21.15.37:0/3344624816 >> 172.21.15.37:0/3344624816 conn(0x7f1bf40b49a0 msgr2=0x7f1bf40b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:06.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.637+0000 7f1bfb19a640 1 -- 172.21.15.37:0/3344624816 shutdown_connections 2024-03-25T18:15:06.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.637+0000 7f1bfb19a640 1 -- 172.21.15.37:0/3344624816 wait complete. 2024-03-25T18:15:06.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:06.637+0000 7f1bfb19a640 1 librados: shutdown 2024-03-25T18:15:06.650 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:10.563 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:07:45.908 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:10.582 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:07:45.928 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:10.615 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:07:45.960 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:10.626 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:07:45.939 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:10.632 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:07:45.977 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:10.634 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:07:45.947 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:10.639 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:07:45.985 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:10.676 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:07:46.021 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:10.677 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:07:45.990 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:13.457 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:13.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.575+0000 7f9458f00640 10 monclient: get_monmap_and_config 2024-03-25T18:15:13.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.575+0000 7f9458f00640 10 monclient: build_initial_monmap 2024-03-25T18:15:13.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.575+0000 7f9458f00640 10 monclient: monmap: 2024-03-25T18:15:13.577 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:13.577 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:13.577 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:15:13.576743+0000 2024-03-25T18:15:13.577 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:15:13.576743+0000 2024-03-25T18:15:13.577 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:13.577 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:13.577 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:13.578 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:15:13.578 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:15:13.578 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:13.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.576+0000 7f9458f00640 1 Processor -- start 2024-03-25T18:15:13.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.576+0000 7f9458f00640 1 -- start start 2024-03-25T18:15:13.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.576+0000 7f9458f00640 10 monclient: init 2024-03-25T18:15:13.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.577+0000 7f9458f00640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:13.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.577+0000 7f9458f00640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:13.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.577+0000 7f9458f00640 10 monclient(hunting): picked mon.noname-b con 0x7f94540a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:13.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.577+0000 7f9458f00640 10 monclient(hunting): picked mon.noname-a con 0x7f945413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:13.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.577+0000 7f9458f00640 10 monclient(hunting): picked mon.noname-c con 0x7f94541430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:13.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.577+0000 7f9458f00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9454142520 con 0x7f945413f710 2024-03-25T18:15:13.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.577+0000 7f9458f00640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94540b28a0 con 0x7f94541430c0 2024-03-25T18:15:13.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.577+0000 7f9458f00640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94540b2a30 con 0x7f94540a98f0 2024-03-25T18:15:13.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.577+0000 7f9458f00640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:13.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.577+0000 7f9458f00640 10 monclient(hunting): authenticate will time out at 7268.829708s 2024-03-25T18:15:13.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.585+0000 7f944bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f945413f710 0x7f94541429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41536/0 (socket says 172.21.15.37:41536) 2024-03-25T18:15:13.586 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.585+0000 7f944bfff640 1 -- 172.21.15.37:0/2517632630 learned_addr learned my addr 172.21.15.37:0/2517632630 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:13.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.600+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3029673687 0 0) 0x7f94540b28a0 con 0x7f94541430c0 2024-03-25T18:15:13.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.600+0000 7f9451d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:13.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.600+0000 7f9451d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:13.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.600+0000 7f9451d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:13.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.600+0000 7f9451d9b640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:15:13.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.600+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9440003520 con 0x7f94541430c0 2024-03-25T18:15:13.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.606+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2660161890 0 0) 0x7f9454142520 con 0x7f945413f710 2024-03-25T18:15:13.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.606+0000 7f9451d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:13.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.606+0000 7f9451d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:13.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.606+0000 7f9451d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:13.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.606+0000 7f9451d9b640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:15:13.607 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.606+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94540b28a0 con 0x7f945413f710 2024-03-25T18:15:13.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.609+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193485262 0 0) 0x7f94540b2a30 con 0x7f94540a98f0 2024-03-25T18:15:13.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.609+0000 7f9451d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:15:13.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.609+0000 7f9451d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:13.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.609+0000 7f9451d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:13.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.609+0000 7f9451d9b640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:15:13.611 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.609+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9454142520 con 0x7f94540a98f0 2024-03-25T18:15:13.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3855215125 0 0) 0x7f9440003520 con 0x7f94541430c0 2024-03-25T18:15:13.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 >> v1:172.21.15.37:6790/0 conn(0x7f94540a98f0 legacy=0x7f94540a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:13.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 >> v1:172.21.15.37:6789/0 conn(0x7f945413f710 legacy=0x7f94541429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:13.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:15:13.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 1 monclient: found mon.noname-c 2024-03-25T18:15:13.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:13.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:15:13.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94540b44c0 con 0x7f94541430c0 2024-03-25T18:15:13.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 10 monclient: _finish_auth 0 2024-03-25T18:15:13.630 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f944c001a10 con 0x7f94541430c0 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9451d9b640 10 monclient: dump: 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:13.631 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:13.632 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:13.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9458f00640 5 monclient: authenticate success, global_id 4561 2024-03-25T18:15:13.632 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.628+0000 7f9458f00640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:13.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.642+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f944c001b70 con 0x7f94541430c0 2024-03-25T18:15:13.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.642+0000 7f9451d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:13.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.642+0000 7f9451d9b640 1 -- 172.21.15.37:0/2517632630 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f944c001f80 con 0x7f94541430c0 2024-03-25T18:15:13.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.642+0000 7f9451d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.642+0000 7f9451d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.642+0000 7f9451d9b640 10 monclient: dump: 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.642+0000 7f9458f00640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:13.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.642+0000 7f9458f00640 4 set_mon_vals no callback set 2024-03-25T18:15:13.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.643+0000 7f9458f00640 10 monclient: shutdown 2024-03-25T18:15:13.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.643+0000 7f9458f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:13.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.643+0000 7f9458f00640 1 -- 172.21.15.37:0/2517632630 >> v1:172.21.15.117:6789/0 conn(0x7f94541430c0 legacy=0x7f945414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:13.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.643+0000 7f9458f00640 1 -- 172.21.15.37:0/2517632630 shutdown_connections 2024-03-25T18:15:13.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.643+0000 7f9458f00640 1 -- 172.21.15.37:0/2517632630 >> 172.21.15.37:0/2517632630 conn(0x7f94540b7ea0 msgr2=0x7f94540b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:13.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.643+0000 7f9458f00640 1 -- 172.21.15.37:0/2517632630 shutdown_connections 2024-03-25T18:15:13.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.643+0000 7f9458f00640 1 -- 172.21.15.37:0/2517632630 wait complete. 2024-03-25T18:15:13.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.644+0000 7f9458f00640 10 monclient: build_initial_monmap 2024-03-25T18:15:13.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.644+0000 7f9458f00640 10 monclient: monmap: 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.644+0000 7f9458f00640 1 librados: starting msgr at 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.644+0000 7f9458f00640 1 librados: starting objecter 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.644+0000 7f9458f00640 1 Processor -- start 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.645+0000 7f9458f00640 1 -- start start 2024-03-25T18:15:13.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.645+0000 7f9458f00640 1 librados: setting wanted keys 2024-03-25T18:15:13.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.645+0000 7f9458f00640 1 librados: calling monclient init 2024-03-25T18:15:13.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.645+0000 7f9458f00640 10 monclient: init 2024-03-25T18:15:13.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.645+0000 7f9458f00640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:13.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.645+0000 7f9458f00640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:13.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.645+0000 7f9458f00640 10 monclient(hunting): picked mon.noname-c con 0x7f94540a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:13.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.645+0000 7f9458f00640 10 monclient(hunting): picked mon.noname-a con 0x7f945413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:13.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.645+0000 7f9458f00640 10 monclient(hunting): picked mon.noname-b con 0x7f94541430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:13.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.645+0000 7f9458f00640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94540abd30 con 0x7f945413f710 2024-03-25T18:15:13.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.646+0000 7f9458f00640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94541d7810 con 0x7f94541430c0 2024-03-25T18:15:13.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.646+0000 7f9458f00640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94541d8440 con 0x7f94540a98f0 2024-03-25T18:15:13.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.646+0000 7f9458f00640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:13.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.646+0000 7f9458f00640 10 monclient(hunting): authenticate will time out at 7268.898442s 2024-03-25T18:15:13.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.655+0000 7f944bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f945413f710 0x7f94541d48f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:41588/0 (socket says 172.21.15.37:41588) 2024-03-25T18:15:13.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.655+0000 7f944bfff640 1 -- 172.21.15.37:0/8215540 learned_addr learned my addr 172.21.15.37:0/8215540 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:13.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.656+0000 7f944affd640 10 client.?.objecter ms_handle_connect 0x7f945413f710 2024-03-25T18:15:13.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.656+0000 7f944affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:13.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.661+0000 7f944affd640 10 client.?.objecter ms_handle_connect 0x7f94541430c0 2024-03-25T18:15:13.663 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.661+0000 7f944affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:13.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.664+0000 7f944affd640 10 client.?.objecter ms_handle_connect 0x7f94540a98f0 2024-03-25T18:15:13.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.664+0000 7f944affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:13.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.667+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2377605422 0 0) 0x7f94541d7810 con 0x7f94541430c0 2024-03-25T18:15:13.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.667+0000 7f944affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:13.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.667+0000 7f944affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:13.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.667+0000 7f944affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:13.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.667+0000 7f944affd640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:15:13.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.667+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9434004eb0 con 0x7f94541430c0 2024-03-25T18:15:13.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.675+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1335845831 0 0) 0x7f94540abd30 con 0x7f945413f710 2024-03-25T18:15:13.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.675+0000 7f944affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:13.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.675+0000 7f944affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:13.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.675+0000 7f944affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:13.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.675+0000 7f944affd640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:15:13.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.675+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94541d7810 con 0x7f945413f710 2024-03-25T18:15:13.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.680+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 930319487 0 0) 0x7f94541d8440 con 0x7f94540a98f0 2024-03-25T18:15:13.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.680+0000 7f944affd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:13.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.680+0000 7f944affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:13.681 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.680+0000 7f944affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:13.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.680+0000 7f944affd640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:15:13.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.680+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94540abd30 con 0x7f94540a98f0 2024-03-25T18:15:13.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.692+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2073538298 0 0) 0x7f9434004eb0 con 0x7f94541430c0 2024-03-25T18:15:13.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.692+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f94541d8440 con 0x7f94541430c0 2024-03-25T18:15:13.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.692+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f944c001e80 con 0x7f94541430c0 2024-03-25T18:15:13.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.692+0000 7f944affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:13.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.692+0000 7f944affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:13.694 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.692+0000 7f944affd640 10 monclient(hunting): dump: 2024-03-25T18:15:13.694 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:13.694 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:13.694 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:13.694 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:13.694 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:13.694 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:13.695 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:13.695 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:13.695 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:13.695 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:13.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.692+0000 7f944affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:13.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.694+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 395706792 0 0) 0x7f94541d7810 con 0x7f945413f710 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.694+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9434004eb0 con 0x7f945413f710 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.694+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f943c001970 con 0x7f945413f710 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.694+0000 7f944affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.694+0000 7f944affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.694+0000 7f944affd640 10 monclient(hunting): dump: 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:13.696 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:13.697 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:13.697 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:13.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 392013519 0 0) 0x7f94541d8440 con 0x7f94541430c0 2024-03-25T18:15:13.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 >> v1:172.21.15.37:6790/0 conn(0x7f94540a98f0 legacy=0x7f94541d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:13.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 >> v1:172.21.15.37:6789/0 conn(0x7f945413f710 legacy=0x7f94541d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:13.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f944affd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:13.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f944affd640 1 monclient: found mon.b 2024-03-25T18:15:13.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f944affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:13.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f944affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:13.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94541d5000 con 0x7f94541430c0 2024-03-25T18:15:13.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f944affd640 10 monclient: _finish_auth 0 2024-03-25T18:15:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f944affd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f944affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f9458f00640 5 monclient: authenticate success, global_id 4564 2024-03-25T18:15:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f9458f00640 10 monclient: _renew_subs 2024-03-25T18:15:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f9458f00640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f9458f00640 1 -- 172.21.15.37:0/8215540 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94541d5250 con 0x7f94541430c0 2024-03-25T18:15:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f9458f00640 10 client.4564.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f9458f00640 10 monclient: _renew_subs 2024-03-25T18:15:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f9458f00640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f9458f00640 1 -- 172.21.15.37:0/8215540 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94541d43c0 con 0x7f94541430c0 2024-03-25T18:15:13.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.702+0000 7f9458f00640 1 librados: init done 2024-03-25T18:15:13.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.703+0000 7f9458f00640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:13.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.703+0000 7f9458f00640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:13.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.703+0000 7f9458f00640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:13.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.703+0000 7f9458f00640 1 -- 172.21.15.37:0/8215540 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94541d7670 con 0x7f94541430c0 2024-03-25T18:15:13.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.979+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f944c002020 con 0x7f94541430c0 2024-03-25T18:15:13.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.979+0000 7f944affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:13.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.979+0000 7f945159a640 4 set_mon_vals no callback set 2024-03-25T18:15:13.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.979+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f944c003c20 con 0x7f94541430c0 2024-03-25T18:15:13.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.979+0000 7f944affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:13.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.979+0000 7f944affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:13.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.979+0000 7f944affd640 10 monclient: dump: 2024-03-25T18:15:13.981 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:13.981 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:13.981 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:13.981 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:13.982 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:13.982 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:13.982 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:13.982 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:13.982 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:13.982 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:13.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.980+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f944c019bf0 con 0x7f94541430c0 2024-03-25T18:15:13.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.981+0000 7f945259c640 10 monclient: get_auth_request con 0x7f943406df70 auth_method 0 2024-03-25T18:15:13.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.981+0000 7f944affd640 10 client.4564.objecter ms_handle_connect 0x7f943406df70 2024-03-25T18:15:13.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.988+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f944c0537a0 con 0x7f94541430c0 2024-03-25T18:15:13.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.988+0000 7f944affd640 10 client.4564.objecter ms_dispatch 0x7f9454000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:13.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.988+0000 7f944affd640 3 client.4564.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:13.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.988+0000 7f944affd640 3 client.4564.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:13.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.988+0000 7f944affd640 20 client.4564.objecter dump_active .. 0 homeless 2024-03-25T18:15:13.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.989+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f944c04f0e0 con 0x7f94541430c0 2024-03-25T18:15:13.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.989+0000 7f944affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:13.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:13.989+0000 7f944affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:14.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.212+0000 7f9458f00640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:14.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.212+0000 7f9458f00640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:14.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.212+0000 7f9458f00640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:14.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.212+0000 7f9458f00640 1 -- 172.21.15.37:0/8215540 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94541d7a50 con 0x7f94541430c0 2024-03-25T18:15:14.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.240+0000 7f944affd640 1 -- 172.21.15.37:0/8215540 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f94541d7a50 con 0x7f94541430c0 2024-03-25T18:15:14.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.240+0000 7f944affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:14.241 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.240+0000 7f944affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:14.241 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:15:14.242 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:14.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.242+0000 7f9458f00640 10 librados: watch_flush enter 2024-03-25T18:15:14.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.242+0000 7f9458f00640 10 librados: watch_flush exit 2024-03-25T18:15:14.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.242+0000 7f9458f00640 20 client.4564.objecter shutdown clearing up homeless session... 2024-03-25T18:15:14.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.242+0000 7f9458f00640 10 client.4564.objecter successfully canceled tick 2024-03-25T18:15:14.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.243+0000 7f9458f00640 1 -- 172.21.15.37:0/8215540 >> v1:172.21.15.117:6800/3703993921 conn(0x7f943406df70 legacy=0x7f9434070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:14.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.243+0000 7f9458f00640 10 monclient: shutdown 2024-03-25T18:15:14.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.243+0000 7f9458f00640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:14.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.243+0000 7f9458f00640 1 -- 172.21.15.37:0/8215540 >> v1:172.21.15.117:6789/0 conn(0x7f94541430c0 legacy=0x7f94541383a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:14.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.243+0000 7f9458f00640 1 -- 172.21.15.37:0/8215540 shutdown_connections 2024-03-25T18:15:14.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.243+0000 7f9458f00640 1 -- 172.21.15.37:0/8215540 >> 172.21.15.37:0/8215540 conn(0x7f94540b7ea0 msgr2=0x7f9454074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:14.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.243+0000 7f9458f00640 1 -- 172.21.15.37:0/8215540 shutdown_connections 2024-03-25T18:15:14.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.243+0000 7f9458f00640 1 -- 172.21.15.37:0/8215540 wait complete. 2024-03-25T18:15:14.245 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:14.243+0000 7f9458f00640 1 librados: shutdown 2024-03-25T18:15:14.258 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:21.064 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:21.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.174+0000 7f09fe70e640 10 monclient: get_monmap_and_config 2024-03-25T18:15:21.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.175+0000 7f09fe70e640 10 monclient: build_initial_monmap 2024-03-25T18:15:21.176 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.175+0000 7f09fe70e640 10 monclient: monmap: 2024-03-25T18:15:21.176 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:21.176 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:21.176 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:15:21.175922+0000 2024-03-25T18:15:21.176 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:15:21.175922+0000 2024-03-25T18:15:21.176 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:21.176 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:21.176 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:21.177 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:15:21.177 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:15:21.177 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:21.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.175+0000 7f09fe70e640 1 Processor -- start 2024-03-25T18:15:21.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.175+0000 7f09fe70e640 1 -- start start 2024-03-25T18:15:21.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.175+0000 7f09fe70e640 10 monclient: init 2024-03-25T18:15:21.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.176+0000 7f09fe70e640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:21.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.176+0000 7f09fe70e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:21.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.176+0000 7f09fe70e640 10 monclient(hunting): picked mon.noname-c con 0x7f09f80a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:21.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.176+0000 7f09fe70e640 10 monclient(hunting): picked mon.noname-a con 0x7f09f813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:21.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.176+0000 7f09fe70e640 10 monclient(hunting): picked mon.noname-b con 0x7f09f81430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:21.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.176+0000 7f09fe70e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f8142520 con 0x7f09f813f710 2024-03-25T18:15:21.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.176+0000 7f09fe70e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f80b28a0 con 0x7f09f80a98f0 2024-03-25T18:15:21.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.176+0000 7f09fe70e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f80b2a30 con 0x7f09f81430c0 2024-03-25T18:15:21.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.176+0000 7f09fe70e640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:21.179 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.176+0000 7f09fe70e640 10 monclient(hunting): authenticate will time out at 7276.428869s 2024-03-25T18:15:21.184 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.184+0000 7f09f7fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f09f80a98f0 0x7f09f80a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:38934/0 (socket says 172.21.15.37:38934) 2024-03-25T18:15:21.185 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.184+0000 7f09f7fff640 1 -- 172.21.15.37:0/1227247002 learned_addr learned my addr 172.21.15.37:0/1227247002 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:21.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.191+0000 7f09f6ffd640 1 -- 172.21.15.37:0/1227247002 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044561271 0 0) 0x7f09f80b28a0 con 0x7f09f80a98f0 2024-03-25T18:15:21.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.191+0000 7f09f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:21.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.191+0000 7f09f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:21.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.191+0000 7f09f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:21.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.191+0000 7f09f6ffd640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:15:21.193 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.191+0000 7f09f6ffd640 1 -- 172.21.15.37:0/1227247002 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e0003520 con 0x7f09f80a98f0 2024-03-25T18:15:21.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.205+0000 7f09f6ffd640 1 -- 172.21.15.37:0/1227247002 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2129421688 0 0) 0x7f09f8142520 con 0x7f09f813f710 2024-03-25T18:15:21.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.205+0000 7f09f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:21.206 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.205+0000 7f09f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:21.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.205+0000 7f09f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:21.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.205+0000 7f09f6ffd640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:15:21.207 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.205+0000 7f09f6ffd640 1 -- 172.21.15.37:0/1227247002 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f80b28a0 con 0x7f09f813f710 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 1 -- 172.21.15.37:0/1227247002 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4066099761 0 0) 0x7f09e0003520 con 0x7f09f80a98f0 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 1 -- 172.21.15.37:0/1227247002 >> v1:172.21.15.37:6790/0 conn(0x7f09f81430c0 legacy=0x7f09f814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 1 -- 172.21.15.37:0/1227247002 >> v1:172.21.15.37:6789/0 conn(0x7f09f813f710 legacy=0x7f09f81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 1 -- 172.21.15.37:0/1227247002 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f80b44c0 con 0x7f09f80a98f0 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:21.208 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 1 -- 172.21.15.37:0/1227247002 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f09e8001690 con 0x7f09f80a98f0 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09f6ffd640 10 monclient: dump: 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09fe70e640 5 monclient: authenticate success, global_id 4567 2024-03-25T18:15:21.209 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.206+0000 7f09fe70e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.211+0000 7f09f6ffd640 1 -- 172.21.15.37:0/1227247002 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09e80017f0 con 0x7f09f80a98f0 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.211+0000 7f09f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.211+0000 7f09f6ffd640 1 -- 172.21.15.37:0/1227247002 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f09e8001c00 con 0x7f09f80a98f0 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.211+0000 7f09f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.211+0000 7f09f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.211+0000 7f09f6ffd640 10 monclient: dump: 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:21.212 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:21.213 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:21.213 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:21.213 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:21.213 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:21.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.211+0000 7f09fe70e640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:21.213 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.211+0000 7f09fe70e640 4 set_mon_vals no callback set 2024-03-25T18:15:21.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.212+0000 7f09fe70e640 10 monclient: shutdown 2024-03-25T18:15:21.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.212+0000 7f09fe70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:21.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.212+0000 7f09fe70e640 1 -- 172.21.15.37:0/1227247002 >> v1:172.21.15.117:6789/0 conn(0x7f09f80a98f0 legacy=0x7f09f80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:21.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.212+0000 7f09fe70e640 1 -- 172.21.15.37:0/1227247002 shutdown_connections 2024-03-25T18:15:21.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.212+0000 7f09fe70e640 1 -- 172.21.15.37:0/1227247002 >> 172.21.15.37:0/1227247002 conn(0x7f09f80b7ea0 msgr2=0x7f09f80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:21.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.212+0000 7f09fe70e640 1 -- 172.21.15.37:0/1227247002 shutdown_connections 2024-03-25T18:15:21.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.212+0000 7f09fe70e640 1 -- 172.21.15.37:0/1227247002 wait complete. 2024-03-25T18:15:21.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.213+0000 7f09fe70e640 10 monclient: build_initial_monmap 2024-03-25T18:15:21.214 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.213+0000 7f09fe70e640 10 monclient: monmap: 2024-03-25T18:15:21.214 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:21.215 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:21.215 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:15:21.215 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:15:21.215 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:21.215 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:15:21.215 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:21.215 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:15:21.215 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:15:21.215 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:21.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.213+0000 7f09fe70e640 1 librados: starting msgr at 2024-03-25T18:15:21.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.213+0000 7f09fe70e640 1 librados: starting objecter 2024-03-25T18:15:21.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.213+0000 7f09fe70e640 1 Processor -- start 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 1 -- start start 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 1 librados: setting wanted keys 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 1 librados: calling monclient init 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 10 monclient: init 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 10 monclient(hunting): picked mon.noname-b con 0x7f09f80a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 10 monclient(hunting): picked mon.noname-a con 0x7f09f813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 10 monclient(hunting): picked mon.noname-c con 0x7f09f81430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f8137ab0 con 0x7f09f813f710 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f8138bd0 con 0x7f09f80a98f0 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09f8138e40 con 0x7f09f81430c0 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:21.217 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.214+0000 7f09fe70e640 10 monclient(hunting): authenticate will time out at 7276.467255s 2024-03-25T18:15:21.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.223+0000 7f09f7fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f09f80a98f0 0x7f09f8137f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:38958/0 (socket says 172.21.15.37:38958) 2024-03-25T18:15:21.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.223+0000 7f09f7fff640 1 -- 172.21.15.37:0/741181813 learned_addr learned my addr 172.21.15.37:0/741181813 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:21.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.233+0000 7f09f4ff9640 10 client.?.objecter ms_handle_connect 0x7f09f80a98f0 2024-03-25T18:15:21.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.233+0000 7f09f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:21.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.238+0000 7f09f4ff9640 10 client.?.objecter ms_handle_connect 0x7f09f81430c0 2024-03-25T18:15:21.239 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.238+0000 7f09f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:21.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.241+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053946995 0 0) 0x7f09f8138bd0 con 0x7f09f80a98f0 2024-03-25T18:15:21.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.241+0000 7f09f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:21.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.241+0000 7f09f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:21.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.241+0000 7f09f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:21.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.241+0000 7f09f4ff9640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:15:21.243 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.241+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e4004eb0 con 0x7f09f80a98f0 2024-03-25T18:15:21.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.245+0000 7f09f4ff9640 10 client.?.objecter ms_handle_connect 0x7f09f813f710 2024-03-25T18:15:21.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.245+0000 7f09f4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:21.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.245+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2017879837 0 0) 0x7f09f8138e40 con 0x7f09f81430c0 2024-03-25T18:15:21.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.245+0000 7f09f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:21.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.245+0000 7f09f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:21.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.245+0000 7f09f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:21.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.245+0000 7f09f4ff9640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:15:21.247 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.245+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f8138bd0 con 0x7f09f81430c0 2024-03-25T18:15:21.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.257+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 965340340 0 0) 0x7f09e4004eb0 con 0x7f09f80a98f0 2024-03-25T18:15:21.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.257+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09f8138e40 con 0x7f09f80a98f0 2024-03-25T18:15:21.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.257+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f09e8001bc0 con 0x7f09f80a98f0 2024-03-25T18:15:21.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.257+0000 7f09f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:21.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.257+0000 7f09f4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:21.258 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.257+0000 7f09f4ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:21.258 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:21.258 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:21.258 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:21.258 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:21.258 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:21.259 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:21.259 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:21.259 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:21.259 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:21.259 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:21.259 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.257+0000 7f09f4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.261+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2971869285 0 0) 0x7f09f8138bd0 con 0x7f09f81430c0 2024-03-25T18:15:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.261+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f09e4004eb0 con 0x7f09f81430c0 2024-03-25T18:15:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.261+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f09f0001b10 con 0x7f09f81430c0 2024-03-25T18:15:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.261+0000 7f09f4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.261+0000 7f09f4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:21.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.261+0000 7f09f4ff9640 10 monclient(hunting): dump: 2024-03-25T18:15:21.263 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:21.263 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:21.263 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:21.263 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:21.263 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:21.263 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:21.263 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:21.263 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:21.263 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:21.263 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:21.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.263+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1001147343 0 0) 0x7f09f8137ab0 con 0x7f09f813f710 2024-03-25T18:15:21.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.263+0000 7f09f4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:15:21.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.263+0000 7f09f4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:21.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.263+0000 7f09f4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:21.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.263+0000 7f09f4ff9640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:15:21.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.263+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f8138bd0 con 0x7f09f813f710 2024-03-25T18:15:21.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.275+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1751371599 0 0) 0x7f09e4004eb0 con 0x7f09f81430c0 2024-03-25T18:15:21.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 >> v1:172.21.15.117:6789/0 conn(0x7f09f80a98f0 legacy=0x7f09f8137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:21.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 >> v1:172.21.15.37:6789/0 conn(0x7f09f813f710 legacy=0x7f09f81df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:21.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09f4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:15:21.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09f4ff9640 1 monclient: found mon.c 2024-03-25T18:15:21.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09f4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09f4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f8139120 con 0x7f09f81430c0 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09f4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09f4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09f4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09fe70e640 5 monclient: authenticate success, global_id 4565 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09fe70e640 10 monclient: _renew_subs 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09fe70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09fe70e640 1 -- 172.21.15.37:0/741181813 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f8139400 con 0x7f09f81430c0 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09fe70e640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09fe70e640 10 monclient: _renew_subs 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09fe70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09fe70e640 1 -- 172.21.15.37:0/741181813 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f80b7880 con 0x7f09f81430c0 2024-03-25T18:15:21.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.276+0000 7f09fe70e640 1 librados: init done 2024-03-25T18:15:21.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.277+0000 7f09fe70e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:21.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.277+0000 7f09fe70e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:21.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.277+0000 7f09fe70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:21.279 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.277+0000 7f09fe70e640 1 -- 172.21.15.37:0/741181813 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f8137810 con 0x7f09f81430c0 2024-03-25T18:15:21.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.596+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09f0002000 con 0x7f09f81430c0 2024-03-25T18:15:21.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.596+0000 7f09f4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:21.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.596+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f09f0003b30 con 0x7f09f81430c0 2024-03-25T18:15:21.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.596+0000 7f09f4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:21.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.596+0000 7f09f4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:21.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.596+0000 7f09f4ff9640 10 monclient: dump: 2024-03-25T18:15:21.597 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:21.597 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:21.597 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:21.598 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:21.598 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:21.598 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:21.598 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:21.598 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:21.598 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:21.598 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:21.598 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.596+0000 7f09f67fc640 4 set_mon_vals no callback set 2024-03-25T18:15:21.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.596+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f09f0003d50 con 0x7f09f81430c0 2024-03-25T18:15:21.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.597+0000 7f09f7fff640 10 monclient: get_auth_request con 0x7f09e406df30 auth_method 0 2024-03-25T18:15:21.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.597+0000 7f09f4ff9640 10 client.4565.objecter ms_handle_connect 0x7f09e406df30 2024-03-25T18:15:21.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.603+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f09f0053810 con 0x7f09f81430c0 2024-03-25T18:15:21.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.603+0000 7f09f4ff9640 10 client.4565.objecter ms_dispatch 0x7f09f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:21.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.603+0000 7f09f4ff9640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:21.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.603+0000 7f09f4ff9640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:21.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.603+0000 7f09f4ff9640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-25T18:15:21.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.603+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09f004f320 con 0x7f09f81430c0 2024-03-25T18:15:21.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.603+0000 7f09f4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:21.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.603+0000 7f09f4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:21.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.844+0000 7f09fe70e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.844+0000 7f09fe70e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.844+0000 7f09fe70e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:21.846 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.844+0000 7f09fe70e640 1 -- 172.21.15.37:0/741181813 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f09f80627b0 con 0x7f09f81430c0 2024-03-25T18:15:21.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.892+0000 7f09f4ff9640 1 -- 172.21.15.37:0/741181813 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f09f0051250 con 0x7f09f81430c0 2024-03-25T18:15:21.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.892+0000 7f09f4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:21.893 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.892+0000 7f09f4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:21.893 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:15:21.893 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:21.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.894+0000 7f09fe70e640 10 librados: watch_flush enter 2024-03-25T18:15:21.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.894+0000 7f09fe70e640 10 librados: watch_flush exit 2024-03-25T18:15:21.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.894+0000 7f09fe70e640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-25T18:15:21.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.894+0000 7f09fe70e640 10 client.4565.objecter successfully canceled tick 2024-03-25T18:15:21.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.894+0000 7f09fe70e640 1 -- 172.21.15.37:0/741181813 >> v1:172.21.15.117:6800/3703993921 conn(0x7f09e406df30 legacy=0x7f09e40703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:21.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.894+0000 7f09fe70e640 10 monclient: shutdown 2024-03-25T18:15:21.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.894+0000 7f09fe70e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:21.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.894+0000 7f09fe70e640 1 -- 172.21.15.37:0/741181813 >> v1:172.21.15.37:6790/0 conn(0x7f09f81430c0 legacy=0x7f09f81e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:21.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.894+0000 7f09fe70e640 1 -- 172.21.15.37:0/741181813 shutdown_connections 2024-03-25T18:15:21.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.894+0000 7f09fe70e640 1 -- 172.21.15.37:0/741181813 >> 172.21.15.37:0/741181813 conn(0x7f09f80b7ea0 msgr2=0x7f09f8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:21.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.895+0000 7f09fe70e640 1 -- 172.21.15.37:0/741181813 shutdown_connections 2024-03-25T18:15:21.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.895+0000 7f09fe70e640 1 -- 172.21.15.37:0/741181813 wait complete. 2024-03-25T18:15:21.897 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:21.895+0000 7f09fe70e640 1 librados: shutdown 2024-03-25T18:15:21.907 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:28.716 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:28.831 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.830+0000 7f93f9e06640 10 monclient: get_monmap_and_config 2024-03-25T18:15:28.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.830+0000 7f93f9e06640 10 monclient: build_initial_monmap 2024-03-25T18:15:28.832 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.830+0000 7f93f9e06640 10 monclient: monmap: 2024-03-25T18:15:28.832 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:28.832 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:28.832 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:15:28.831469+0000 2024-03-25T18:15:28.832 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:15:28.831469+0000 2024-03-25T18:15:28.832 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:28.832 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:28.832 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:28.832 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:15:28.833 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:15:28.833 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:28.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.831+0000 7f93f9e06640 1 Processor -- start 2024-03-25T18:15:28.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.831+0000 7f93f9e06640 1 -- start start 2024-03-25T18:15:28.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.831+0000 7f93f9e06640 10 monclient: init 2024-03-25T18:15:28.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.832+0000 7f93f9e06640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:28.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.832+0000 7f93f9e06640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:28.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.832+0000 7f93f9e06640 10 monclient(hunting): picked mon.noname-a con 0x7f93f40a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:28.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.832+0000 7f93f9e06640 10 monclient(hunting): picked mon.noname-b con 0x7f93f413f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:28.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.832+0000 7f93f9e06640 10 monclient(hunting): picked mon.noname-c con 0x7f93f41430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:28.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.832+0000 7f93f9e06640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f4142520 con 0x7f93f40a98f0 2024-03-25T18:15:28.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.832+0000 7f93f9e06640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f40b28a0 con 0x7f93f41430c0 2024-03-25T18:15:28.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.832+0000 7f93f9e06640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f40b2a30 con 0x7f93f413f710 2024-03-25T18:15:28.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.832+0000 7f93f9e06640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:28.836 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.832+0000 7f93f9e06640 10 monclient(hunting): authenticate will time out at 7284.085078s 2024-03-25T18:15:28.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.843+0000 7f93f3fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f93f41430c0 0x7f93f414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:47254/0 (socket says 172.21.15.37:47254) 2024-03-25T18:15:28.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.843+0000 7f93f3fff640 1 -- 172.21.15.37:0/4178294494 learned_addr learned my addr 172.21.15.37:0/4178294494 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:28.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.866+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 590174079 0 0) 0x7f93f4142520 con 0x7f93f40a98f0 2024-03-25T18:15:28.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.866+0000 7f93f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.866+0000 7f93f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.866+0000 7f93f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.866+0000 7f93f27fc640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:15:28.867 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.866+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93dc003520 con 0x7f93f40a98f0 2024-03-25T18:15:28.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.872+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3844192058 0 0) 0x7f93f40b28a0 con 0x7f93f41430c0 2024-03-25T18:15:28.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.872+0000 7f93f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.872+0000 7f93f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.872+0000 7f93f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.872+0000 7f93f27fc640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:15:28.873 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.872+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f4142520 con 0x7f93f41430c0 2024-03-25T18:15:28.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.876+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3607222661 0 0) 0x7f93f40b2a30 con 0x7f93f413f710 2024-03-25T18:15:28.877 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.876+0000 7f93f27fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.876+0000 7f93f27fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.876+0000 7f93f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.876+0000 7f93f27fc640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:15:28.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.876+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f40b28a0 con 0x7f93f413f710 2024-03-25T18:15:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2840665002 0 0) 0x7f93f4142520 con 0x7f93f41430c0 2024-03-25T18:15:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 >> v1:172.21.15.37:6790/0 conn(0x7f93f413f710 legacy=0x7f93f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 >> v1:172.21.15.37:6789/0 conn(0x7f93f40a98f0 legacy=0x7f93f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 1 monclient: found mon.noname-c 2024-03-25T18:15:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:15:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f40b44c0 con 0x7f93f41430c0 2024-03-25T18:15:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:28.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f93ec0019f0 con 0x7f93f41430c0 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f27fc640 10 monclient: dump: 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:28.887 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:28.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f9e06640 5 monclient: authenticate success, global_id 4573 2024-03-25T18:15:28.888 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.884+0000 7f93f9e06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:28.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.909+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93ec001b50 con 0x7f93f41430c0 2024-03-25T18:15:28.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.909+0000 7f93f27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.909+0000 7f93f27fc640 1 -- 172.21.15.37:0/4178294494 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f93ec001f60 con 0x7f93f41430c0 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.909+0000 7f93f9e06640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.909+0000 7f93f9e06640 4 set_mon_vals no callback set 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.909+0000 7f93f27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.909+0000 7f93f27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.909+0000 7f93f27fc640 10 monclient: dump: 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:28.911 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:28.912 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:28.912 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:28.912 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:28.912 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:28.914 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.910+0000 7f93f9e06640 10 monclient: shutdown 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.910+0000 7f93f9e06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.910+0000 7f93f9e06640 1 -- 172.21.15.37:0/4178294494 >> v1:172.21.15.117:6789/0 conn(0x7f93f41430c0 legacy=0x7f93f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.910+0000 7f93f9e06640 1 -- 172.21.15.37:0/4178294494 shutdown_connections 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.910+0000 7f93f9e06640 1 -- 172.21.15.37:0/4178294494 >> 172.21.15.37:0/4178294494 conn(0x7f93f40b7ea0 msgr2=0x7f93f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.910+0000 7f93f9e06640 1 -- 172.21.15.37:0/4178294494 shutdown_connections 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.910+0000 7f93f9e06640 1 -- 172.21.15.37:0/4178294494 wait complete. 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.911+0000 7f93f9e06640 10 monclient: build_initial_monmap 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.911+0000 7f93f9e06640 10 monclient: monmap: 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:15:28.915 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.911+0000 7f93f9e06640 1 librados: starting msgr at 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.911+0000 7f93f9e06640 1 librados: starting objecter 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.911+0000 7f93f9e06640 1 Processor -- start 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.911+0000 7f93f9e06640 1 -- start start 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.911+0000 7f93f9e06640 1 librados: setting wanted keys 2024-03-25T18:15:28.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.911+0000 7f93f9e06640 1 librados: calling monclient init 2024-03-25T18:15:28.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.911+0000 7f93f9e06640 10 monclient: init 2024-03-25T18:15:28.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.912+0000 7f93f9e06640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:28.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.912+0000 7f93f9e06640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:28.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.912+0000 7f93f9e06640 10 monclient(hunting): picked mon.noname-a con 0x7f93f40a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:28.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.912+0000 7f93f9e06640 10 monclient(hunting): picked mon.noname-c con 0x7f93f413f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:28.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.912+0000 7f93f9e06640 10 monclient(hunting): picked mon.noname-b con 0x7f93f41430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:28.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.912+0000 7f93f9e06640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f4137ce0 con 0x7f93f40a98f0 2024-03-25T18:15:28.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.912+0000 7f93f9e06640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f41396e0 con 0x7f93f41430c0 2024-03-25T18:15:28.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.912+0000 7f93f9e06640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93f41399c0 con 0x7f93f413f710 2024-03-25T18:15:28.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.912+0000 7f93f9e06640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:28.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.912+0000 7f93f9e06640 10 monclient(hunting): authenticate will time out at 7284.164853s 2024-03-25T18:15:28.925 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.924+0000 7f93f37fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f93f40a98f0 0x7f93f41da780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:57642/0 (socket says 172.21.15.37:57642) 2024-03-25T18:15:28.925 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.924+0000 7f93f37fe640 1 -- 172.21.15.37:0/638531632 learned_addr learned my addr 172.21.15.37:0/638531632 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:28.926 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.925+0000 7f93dbfff640 10 client.?.objecter ms_handle_connect 0x7f93f40a98f0 2024-03-25T18:15:28.926 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.925+0000 7f93dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:28.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.929+0000 7f93dbfff640 10 client.?.objecter ms_handle_connect 0x7f93f413f710 2024-03-25T18:15:28.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.929+0000 7f93dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:28.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.931+0000 7f93dbfff640 10 client.?.objecter ms_handle_connect 0x7f93f41430c0 2024-03-25T18:15:28.932 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.931+0000 7f93dbfff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:28.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.940+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2228447227 0 0) 0x7f93f4137ce0 con 0x7f93f40a98f0 2024-03-25T18:15:28.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.940+0000 7f93dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.940+0000 7f93dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.940+0000 7f93dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.940+0000 7f93dbfff640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:15:28.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.940+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93e0004eb0 con 0x7f93f40a98f0 2024-03-25T18:15:28.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.945+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845395325 0 0) 0x7f93f41396e0 con 0x7f93f41430c0 2024-03-25T18:15:28.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.945+0000 7f93dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.945+0000 7f93dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.945+0000 7f93dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.945+0000 7f93dbfff640 10 monclient(hunting): my global_id is 4576 2024-03-25T18:15:28.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.945+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f4137ce0 con 0x7f93f41430c0 2024-03-25T18:15:28.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.947+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2961208788 0 0) 0x7f93f41399c0 con 0x7f93f413f710 2024-03-25T18:15:28.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.947+0000 7f93dbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:28.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.947+0000 7f93dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:28.948 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.947+0000 7f93dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:28.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.947+0000 7f93dbfff640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:15:28.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.947+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93f41396e0 con 0x7f93f413f710 2024-03-25T18:15:28.964 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.963+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2780607361 0 0) 0x7f93f4137ce0 con 0x7f93f41430c0 2024-03-25T18:15:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.963+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93f41399c0 con 0x7f93f41430c0 2024-03-25T18:15:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.963+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f93ec001e60 con 0x7f93f41430c0 2024-03-25T18:15:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.963+0000 7f93dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.963+0000 7f93dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:28.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.963+0000 7f93dbfff640 10 monclient(hunting): dump: 2024-03-25T18:15:28.965 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:28.965 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:28.965 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:28.966 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:28.966 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:28.966 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:28.966 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:28.966 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:28.966 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:28.966 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:28.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.963+0000 7f93dbfff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:28.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.968+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 558558622 0 0) 0x7f93e0004eb0 con 0x7f93f40a98f0 2024-03-25T18:15:28.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.968+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f93f4137ce0 con 0x7f93f40a98f0 2024-03-25T18:15:28.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.968+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f93e4001990 con 0x7f93f40a98f0 2024-03-25T18:15:28.969 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.968+0000 7f93dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.968+0000 7f93dbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.968+0000 7f93dbfff640 10 monclient(hunting): dump: 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:28.970 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:28.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 803436520 0 0) 0x7f93f4137ce0 con 0x7f93f40a98f0 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 >> v1:172.21.15.37:6790/0 conn(0x7f93f413f710 legacy=0x7f93f4138320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 >> v1:172.21.15.117:6789/0 conn(0x7f93f41430c0 legacy=0x7f93f41375d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93dbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93dbfff640 1 monclient: found mon.a 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93dbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93f40b0fb0 con 0x7f93f40a98f0 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93dbfff640 10 monclient: _finish_auth 0 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93dbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93f9e06640 5 monclient: authenticate success, global_id 4578 2024-03-25T18:15:28.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93f9e06640 10 monclient: _renew_subs 2024-03-25T18:15:28.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93f9e06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:15:28.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93f9e06640 1 -- 172.21.15.37:0/638531632 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f40afdc0 con 0x7f93f40a98f0 2024-03-25T18:15:28.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93f9e06640 10 client.4578.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:28.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93f9e06640 10 monclient: _renew_subs 2024-03-25T18:15:28.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93f9e06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:15:28.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93f9e06640 1 -- 172.21.15.37:0/638531632 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f40b02f0 con 0x7f93f40a98f0 2024-03-25T18:15:28.981 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.978+0000 7f93f9e06640 1 librados: init done 2024-03-25T18:15:28.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.979+0000 7f93f9e06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:28.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.979+0000 7f93f9e06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:28.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.979+0000 7f93f9e06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:15:28.982 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:28.979+0000 7f93f9e06640 1 -- 172.21.15.37:0/638531632 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f40627b0 con 0x7f93f40a98f0 2024-03-25T18:15:29.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.311+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f93e40035f0 con 0x7f93f40a98f0 2024-03-25T18:15:29.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.311+0000 7f93dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:29.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.311+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f93e40039c0 con 0x7f93f40a98f0 2024-03-25T18:15:29.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.311+0000 7f93dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:29.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.311+0000 7f93dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:29.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.311+0000 7f93dbfff640 10 monclient: dump: 2024-03-25T18:15:29.313 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:29.314 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:29.314 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:29.314 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:29.314 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:29.314 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:29.314 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:29.314 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:29.314 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:29.314 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:29.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.311+0000 7f93f1ffb640 4 set_mon_vals no callback set 2024-03-25T18:15:29.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.312+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f93e4003be0 con 0x7f93f40a98f0 2024-03-25T18:15:29.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.313+0000 7f93f2ffd640 10 monclient: get_auth_request con 0x7f93e006df60 auth_method 0 2024-03-25T18:15:29.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.313+0000 7f93dbfff640 10 client.4578.objecter ms_handle_connect 0x7f93e006df60 2024-03-25T18:15:29.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.319+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f93e40547a0 con 0x7f93f40a98f0 2024-03-25T18:15:29.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.319+0000 7f93dbfff640 10 client.4578.objecter ms_dispatch 0x7f93f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:29.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.319+0000 7f93dbfff640 3 client.4578.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:29.320 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.319+0000 7f93dbfff640 3 client.4578.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:29.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.319+0000 7f93dbfff640 20 client.4578.objecter dump_active .. 0 homeless 2024-03-25T18:15:29.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.319+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f93e40502b0 con 0x7f93f40a98f0 2024-03-25T18:15:29.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.319+0000 7f93dbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:29.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.319+0000 7f93dbfff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:29.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.573+0000 7f93f9e06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:29.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.573+0000 7f93f9e06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:29.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.573+0000 7f93f9e06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:15:29.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.573+0000 7f93f9e06640 1 -- 172.21.15.37:0/638531632 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f93f40b71a0 con 0x7f93f40a98f0 2024-03-25T18:15:29.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.652+0000 7f93dbfff640 1 -- 172.21.15.37:0/638531632 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f93e40521e0 con 0x7f93f40a98f0 2024-03-25T18:15:29.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.652+0000 7f93dbfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:29.654 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.652+0000 7f93dbfff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:29.654 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:15:29.654 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:29.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 10 librados: watch_flush enter 2024-03-25T18:15:29.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 10 librados: watch_flush exit 2024-03-25T18:15:29.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 20 client.4578.objecter shutdown clearing up homeless session... 2024-03-25T18:15:29.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 10 client.4578.objecter successfully canceled tick 2024-03-25T18:15:29.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 1 -- 172.21.15.37:0/638531632 >> v1:172.21.15.117:6800/3703993921 conn(0x7f93e006df60 legacy=0x7f93e0070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:29.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 10 monclient: shutdown 2024-03-25T18:15:29.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:29.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 1 -- 172.21.15.37:0/638531632 >> v1:172.21.15.37:6789/0 conn(0x7f93f40a98f0 legacy=0x7f93f41da780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:29.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 1 -- 172.21.15.37:0/638531632 shutdown_connections 2024-03-25T18:15:29.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 1 -- 172.21.15.37:0/638531632 >> 172.21.15.37:0/638531632 conn(0x7f93f40b7ea0 msgr2=0x7f93f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:29.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 1 -- 172.21.15.37:0/638531632 shutdown_connections 2024-03-25T18:15:29.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 1 -- 172.21.15.37:0/638531632 wait complete. 2024-03-25T18:15:29.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:29.655+0000 7f93f9e06640 1 librados: shutdown 2024-03-25T18:15:29.667 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:30.610 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:08:05.955 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:30.617 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:08:05.963 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:30.654 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:08:05.999 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:30.664 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:08:05.977 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:30.669 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:08:06.014 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:30.672 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:08:05.985 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:30.681 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:08:06.026 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:30.697 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:08:06.010 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:30.715 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:08:06.060 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:36.474 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:36.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.589+0000 7f804475d640 10 monclient: get_monmap_and_config 2024-03-25T18:15:36.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.589+0000 7f804475d640 10 monclient: build_initial_monmap 2024-03-25T18:15:36.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.589+0000 7f804475d640 10 monclient: monmap: 2024-03-25T18:15:36.592 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:36.592 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:36.592 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:15:36.591089+0000 2024-03-25T18:15:36.592 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:15:36.591089+0000 2024-03-25T18:15:36.592 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:36.592 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:36.592 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:36.592 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:15:36.592 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:15:36.592 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:36.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.589+0000 7f804475d640 1 Processor -- start 2024-03-25T18:15:36.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.590+0000 7f804475d640 1 -- start start 2024-03-25T18:15:36.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.590+0000 7f804475d640 10 monclient: init 2024-03-25T18:15:36.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.590+0000 7f804475d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:36.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.590+0000 7f804475d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:36.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.590+0000 7f804475d640 10 monclient(hunting): picked mon.noname-a con 0x7f803c0a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:36.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.591+0000 7f804475d640 10 monclient(hunting): picked mon.noname-c con 0x7f803c13f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:36.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.591+0000 7f804475d640 10 monclient(hunting): picked mon.noname-b con 0x7f803c1430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:36.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.591+0000 7f804475d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803c142520 con 0x7f803c0a98f0 2024-03-25T18:15:36.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.591+0000 7f804475d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803c0b28a0 con 0x7f803c13f710 2024-03-25T18:15:36.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.591+0000 7f804475d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803c0b2a30 con 0x7f803c1430c0 2024-03-25T18:15:36.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.591+0000 7f804475d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:36.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.591+0000 7f804475d640 10 monclient(hunting): authenticate will time out at 7291.844360s 2024-03-25T18:15:36.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.602+0000 7f80424f9640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f803c0a98f0 0x7f803c0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34020/0 (socket says 172.21.15.37:34020) 2024-03-25T18:15:36.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.602+0000 7f80424f9640 1 -- 172.21.15.37:0/2148996497 learned_addr learned my addr 172.21.15.37:0/2148996497 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:36.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.623+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862853393 0 0) 0x7f803c142520 con 0x7f803c0a98f0 2024-03-25T18:15:36.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.623+0000 7f80414f7640 10 monclient(hunting): _negotiate 2024-03-25T18:15:36.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.623+0000 7f80414f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:36.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.623+0000 7f80414f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:36.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.623+0000 7f80414f7640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:15:36.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.623+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8028003520 con 0x7f803c0a98f0 2024-03-25T18:15:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.626+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491251698 0 0) 0x7f803c0b28a0 con 0x7f803c13f710 2024-03-25T18:15:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.626+0000 7f80414f7640 10 monclient(hunting): _negotiate 2024-03-25T18:15:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.626+0000 7f80414f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.626+0000 7f80414f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.626+0000 7f80414f7640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:15:36.628 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.626+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f803c142520 con 0x7f803c13f710 2024-03-25T18:15:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.632+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1500682033 0 0) 0x7f803c0b2a30 con 0x7f803c1430c0 2024-03-25T18:15:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.632+0000 7f80414f7640 10 monclient(hunting): _negotiate 2024-03-25T18:15:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.632+0000 7f80414f7640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.632+0000 7f80414f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.632+0000 7f80414f7640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:15:36.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.632+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f803c0b28a0 con 0x7f803c1430c0 2024-03-25T18:15:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4058679428 0 0) 0x7f803c142520 con 0x7f803c13f710 2024-03-25T18:15:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 >> v1:172.21.15.37:6790/0 conn(0x7f803c1430c0 legacy=0x7f803c14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 >> v1:172.21.15.37:6789/0 conn(0x7f803c0a98f0 legacy=0x7f803c0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:36.642 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 10 monclient: _finish_hunting 0 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 1 monclient: found mon.noname-c 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c0b44c0 con 0x7f803c13f710 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 10 monclient: _finish_auth 0 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 10 monclient: _check_auth_tickets 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f802c001670 con 0x7f803c13f710 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f804475d640 5 monclient: authenticate success, global_id 4579 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f804475d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f80414f7640 10 monclient: dump: 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:36.643 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:36.644 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:36.644 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:36.644 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:36.644 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:36.644 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:36.644 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:36.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.640+0000 7f804475d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:36.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.654+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f802c0017d0 con 0x7f803c13f710 2024-03-25T18:15:36.656 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.654+0000 7f80414f7640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.654+0000 7f80414f7640 1 -- 172.21.15.37:0/2148996497 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f802c001be0 con 0x7f803c13f710 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.654+0000 7f804475d640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.654+0000 7f804475d640 4 set_mon_vals no callback set 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.654+0000 7f80414f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.654+0000 7f80414f7640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.654+0000 7f80414f7640 10 monclient: dump: 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:36.657 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:36.658 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:36.658 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:36.658 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:36.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.655+0000 7f804475d640 10 monclient: shutdown 2024-03-25T18:15:36.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.655+0000 7f804475d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:36.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.655+0000 7f804475d640 1 -- 172.21.15.37:0/2148996497 >> v1:172.21.15.117:6789/0 conn(0x7f803c13f710 legacy=0x7f803c1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:36.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.655+0000 7f804475d640 1 -- 172.21.15.37:0/2148996497 shutdown_connections 2024-03-25T18:15:36.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.655+0000 7f804475d640 1 -- 172.21.15.37:0/2148996497 >> 172.21.15.37:0/2148996497 conn(0x7f803c0b7ea0 msgr2=0x7f803c0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:36.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.655+0000 7f804475d640 1 -- 172.21.15.37:0/2148996497 shutdown_connections 2024-03-25T18:15:36.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.655+0000 7f804475d640 1 -- 172.21.15.37:0/2148996497 wait complete. 2024-03-25T18:15:36.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.656+0000 7f804475d640 10 monclient: build_initial_monmap 2024-03-25T18:15:36.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.656+0000 7f804475d640 10 monclient: monmap: 2024-03-25T18:15:36.660 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.656+0000 7f804475d640 1 librados: starting msgr at 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.656+0000 7f804475d640 1 librados: starting objecter 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.656+0000 7f804475d640 1 Processor -- start 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.656+0000 7f804475d640 1 -- start start 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.656+0000 7f804475d640 1 librados: setting wanted keys 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.656+0000 7f804475d640 1 librados: calling monclient init 2024-03-25T18:15:36.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.656+0000 7f804475d640 10 monclient: init 2024-03-25T18:15:36.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.657+0000 7f804475d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:36.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.657+0000 7f804475d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:36.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.657+0000 7f804475d640 10 monclient(hunting): picked mon.noname-c con 0x7f803c0a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:36.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.657+0000 7f804475d640 10 monclient(hunting): picked mon.noname-a con 0x7f803c13f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:36.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.657+0000 7f804475d640 10 monclient(hunting): picked mon.noname-b con 0x7f803c1430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:36.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.657+0000 7f804475d640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803c1d7490 con 0x7f803c13f710 2024-03-25T18:15:36.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.657+0000 7f804475d640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803c0abb90 con 0x7f803c1430c0 2024-03-25T18:15:36.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.657+0000 7f804475d640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f803c1d76c0 con 0x7f803c0a98f0 2024-03-25T18:15:36.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.657+0000 7f804475d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:36.662 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.657+0000 7f804475d640 10 monclient(hunting): authenticate will time out at 7291.910745s 2024-03-25T18:15:36.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.667+0000 7f8042cfa640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f803c1430c0 0x7f803c133f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:51846/0 (socket says 172.21.15.37:51846) 2024-03-25T18:15:36.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.667+0000 7f8042cfa640 1 -- 172.21.15.37:0/2528883035 learned_addr learned my addr 172.21.15.37:0/2528883035 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:36.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.667+0000 7f8032ffd640 10 client.?.objecter ms_handle_connect 0x7f803c1430c0 2024-03-25T18:15:36.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.667+0000 7f8032ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:36.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.671+0000 7f8032ffd640 10 client.?.objecter ms_handle_connect 0x7f803c0a98f0 2024-03-25T18:15:36.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.671+0000 7f8032ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:36.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.684+0000 7f8032ffd640 10 client.?.objecter ms_handle_connect 0x7f803c13f710 2024-03-25T18:15:36.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.684+0000 7f8032ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:36.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.684+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709169063 0 0) 0x7f803c0abb90 con 0x7f803c1430c0 2024-03-25T18:15:36.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.684+0000 7f8032ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:36.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.684+0000 7f8032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:36.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.684+0000 7f8032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:36.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.684+0000 7f8032ffd640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:15:36.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.684+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8020004eb0 con 0x7f803c1430c0 2024-03-25T18:15:36.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.697+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1958636632 0 0) 0x7f803c1d76c0 con 0x7f803c0a98f0 2024-03-25T18:15:36.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.697+0000 7f8032ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:36.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.697+0000 7f8032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:36.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.697+0000 7f8032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:36.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.697+0000 7f8032ffd640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:15:36.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.697+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f803c0abb90 con 0x7f803c0a98f0 2024-03-25T18:15:36.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.705+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3954182385 0 0) 0x7f803c1d7490 con 0x7f803c13f710 2024-03-25T18:15:36.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.705+0000 7f8032ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:15:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.705+0000 7f8032ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.705+0000 7f8032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.705+0000 7f8032ffd640 10 monclient(hunting): my global_id is 4584 2024-03-25T18:15:36.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.705+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f803c1d76c0 con 0x7f803c13f710 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.713+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4294554849 0 0) 0x7f8020004eb0 con 0x7f803c1430c0 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.713+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f803c1d7490 con 0x7f803c1430c0 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.713+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8034001c70 con 0x7f803c1430c0 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.713+0000 7f8032ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.713+0000 7f8032ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.713+0000 7f8032ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:36.716 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:36.717 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:36.717 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:36.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.713+0000 7f8032ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:36.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.718+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 311510165 0 0) 0x7f803c0abb90 con 0x7f803c0a98f0 2024-03-25T18:15:36.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.718+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8020004eb0 con 0x7f803c0a98f0 2024-03-25T18:15:36.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.718+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8038001950 con 0x7f803c0a98f0 2024-03-25T18:15:36.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.718+0000 7f8032ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.718+0000 7f8032ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.718+0000 7f8032ffd640 10 monclient(hunting): dump: 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:36.721 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:36.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.726+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2350433701 0 0) 0x7f803c1d7490 con 0x7f803c1430c0 2024-03-25T18:15:36.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 >> v1:172.21.15.37:6790/0 conn(0x7f803c0a98f0 legacy=0x7f803c1d7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:36.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 >> v1:172.21.15.37:6789/0 conn(0x7f803c13f710 legacy=0x7f803c138510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:36.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f8032ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:15:36.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f8032ffd640 1 monclient: found mon.b 2024-03-25T18:15:36.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f8032ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f8032ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f803c1d82f0 con 0x7f803c1430c0 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f8032ffd640 10 monclient: _finish_auth 0 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f8032ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f8032ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f804475d640 5 monclient: authenticate success, global_id 4582 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f804475d640 10 monclient: _renew_subs 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f804475d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f804475d640 1 -- 172.21.15.37:0/2528883035 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f803c138c20 con 0x7f803c1430c0 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f804475d640 10 client.4582.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f804475d640 10 monclient: _renew_subs 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f804475d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f804475d640 1 -- 172.21.15.37:0/2528883035 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f803c1379e0 con 0x7f803c1430c0 2024-03-25T18:15:36.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.727+0000 7f804475d640 1 librados: init done 2024-03-25T18:15:36.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.728+0000 7f804475d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:36.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.728+0000 7f804475d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:36.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.728+0000 7f804475d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:36.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.728+0000 7f804475d640 1 -- 172.21.15.37:0/2528883035 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f803c0b3ef0 con 0x7f803c1430c0 2024-03-25T18:15:36.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.995+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8034003990 con 0x7f803c1430c0 2024-03-25T18:15:36.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.995+0000 7f8032ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:36.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.995+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8034003da0 con 0x7f803c1430c0 2024-03-25T18:15:36.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.995+0000 7f8032ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:36.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.995+0000 7f8032ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:36.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.995+0000 7f8032ffd640 10 monclient: dump: 2024-03-25T18:15:36.998 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:36.998 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:36.998 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:36.998 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:36.998 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:36.998 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:36.999 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:36.999 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:36.999 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:36.999 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:36.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.995+0000 7f8040cf6640 4 set_mon_vals no callback set 2024-03-25T18:15:36.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.996+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f8034003fc0 con 0x7f803c1430c0 2024-03-25T18:15:36.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.996+0000 7f80424f9640 10 monclient: get_auth_request con 0x7f802006df70 auth_method 0 2024-03-25T18:15:36.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:36.996+0000 7f8032ffd640 10 client.4582.objecter ms_handle_connect 0x7f802006df70 2024-03-25T18:15:37.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.003+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f8034054ae0 con 0x7f803c1430c0 2024-03-25T18:15:37.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.003+0000 7f8032ffd640 10 client.4582.objecter ms_dispatch 0x7f803c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:37.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.003+0000 7f8032ffd640 3 client.4582.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:37.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.003+0000 7f8032ffd640 3 client.4582.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:37.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.003+0000 7f8032ffd640 20 client.4582.objecter dump_active .. 0 homeless 2024-03-25T18:15:37.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.006+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80340505f0 con 0x7f803c1430c0 2024-03-25T18:15:37.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.006+0000 7f8032ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:37.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.006+0000 7f8032ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:37.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.260+0000 7f804475d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:37.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.260+0000 7f804475d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:37.263 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.260+0000 7f804475d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:37.263 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.260+0000 7f804475d640 1 -- 172.21.15.37:0/2528883035 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f803c14f320 con 0x7f803c1430c0 2024-03-25T18:15:37.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.282+0000 7f8032ffd640 1 -- 172.21.15.37:0/2528883035 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f8034052520 con 0x7f803c1430c0 2024-03-25T18:15:37.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.282+0000 7f8032ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:37.284 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.282+0000 7f8032ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:37.284 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:15:37.285 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:37.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.284+0000 7f804475d640 10 librados: watch_flush enter 2024-03-25T18:15:37.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.284+0000 7f804475d640 10 librados: watch_flush exit 2024-03-25T18:15:37.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.284+0000 7f804475d640 20 client.4582.objecter shutdown clearing up homeless session... 2024-03-25T18:15:37.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.284+0000 7f804475d640 10 client.4582.objecter successfully canceled tick 2024-03-25T18:15:37.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.284+0000 7f804475d640 1 -- 172.21.15.37:0/2528883035 >> v1:172.21.15.117:6800/3703993921 conn(0x7f802006df70 legacy=0x7f8020070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:37.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.284+0000 7f804475d640 10 monclient: shutdown 2024-03-25T18:15:37.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.284+0000 7f804475d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:37.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.284+0000 7f804475d640 1 -- 172.21.15.37:0/2528883035 >> v1:172.21.15.117:6789/0 conn(0x7f803c1430c0 legacy=0x7f803c133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:37.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.285+0000 7f804475d640 1 -- 172.21.15.37:0/2528883035 shutdown_connections 2024-03-25T18:15:37.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.285+0000 7f804475d640 1 -- 172.21.15.37:0/2528883035 >> 172.21.15.37:0/2528883035 conn(0x7f803c0b7ea0 msgr2=0x7f803c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:37.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.285+0000 7f804475d640 1 -- 172.21.15.37:0/2528883035 shutdown_connections 2024-03-25T18:15:37.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.285+0000 7f804475d640 1 -- 172.21.15.37:0/2528883035 wait complete. 2024-03-25T18:15:37.288 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:37.285+0000 7f804475d640 1 librados: shutdown 2024-03-25T18:15:37.299 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:44.105 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:44.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.220+0000 7f91a31a8640 10 monclient: get_monmap_and_config 2024-03-25T18:15:44.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.220+0000 7f91a31a8640 10 monclient: build_initial_monmap 2024-03-25T18:15:44.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.220+0000 7f91a31a8640 10 monclient: monmap: 2024-03-25T18:15:44.221 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:44.221 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:44.221 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:15:44.220967+0000 2024-03-25T18:15:44.221 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:15:44.220967+0000 2024-03-25T18:15:44.221 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:44.221 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:44.222 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:44.222 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:15:44.222 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:15:44.222 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:44.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.220+0000 7f91a31a8640 1 Processor -- start 2024-03-25T18:15:44.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.220+0000 7f91a31a8640 1 -- start start 2024-03-25T18:15:44.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.220+0000 7f91a31a8640 10 monclient: init 2024-03-25T18:15:44.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.221+0000 7f91a31a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:44.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.221+0000 7f91a31a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:44.223 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.221+0000 7f91a31a8640 10 monclient(hunting): picked mon.noname-c con 0x7f919c0b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:44.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.221+0000 7f91a31a8640 10 monclient(hunting): picked mon.noname-a con 0x7f919c0b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:44.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.221+0000 7f91a31a8640 10 monclient(hunting): picked mon.noname-b con 0x7f919c14be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:44.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.221+0000 7f91a31a8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919c0af940 con 0x7f919c0b1730 2024-03-25T18:15:44.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.221+0000 7f91a31a8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919c0b0a70 con 0x7f919c0b4d60 2024-03-25T18:15:44.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.221+0000 7f91a31a8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919c0b0d50 con 0x7f919c14be90 2024-03-25T18:15:44.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.221+0000 7f91a31a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:44.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.221+0000 7f91a31a8640 10 monclient(hunting): authenticate will time out at 7299.474045s 2024-03-25T18:15:44.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.228+0000 7f91a1745640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f919c14be90 0x7f919c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:57638/0 (socket says 172.21.15.37:57638) 2024-03-25T18:15:44.229 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.228+0000 7f91a1745640 1 -- 172.21.15.37:0/2705546414 learned_addr learned my addr 172.21.15.37:0/2705546414 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:44.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.242+0000 7f919b7fe640 1 -- 172.21.15.37:0/2705546414 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 596400566 0 0) 0x7f919c0b0d50 con 0x7f919c14be90 2024-03-25T18:15:44.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.242+0000 7f919b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:44.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.242+0000 7f919b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:44.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.242+0000 7f919b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:44.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.242+0000 7f919b7fe640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:15:44.244 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.242+0000 7f919b7fe640 1 -- 172.21.15.37:0/2705546414 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f918c003520 con 0x7f919c14be90 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 1 -- 172.21.15.37:0/2705546414 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3410844108 0 0) 0x7f918c003520 con 0x7f919c14be90 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 1 -- 172.21.15.37:0/2705546414 >> v1:172.21.15.117:6789/0 conn(0x7f919c0b4d60 legacy=0x7f919c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 1 -- 172.21.15.37:0/2705546414 >> v1:172.21.15.37:6789/0 conn(0x7f919c0b1730 legacy=0x7f919c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 1 monclient: found mon.noname-b 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 1 -- 172.21.15.37:0/2705546414 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c0b1030 con 0x7f919c14be90 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 1 -- 172.21.15.37:0/2705546414 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9194001a10 con 0x7f919c14be90 2024-03-25T18:15:44.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f91a31a8640 5 monclient: authenticate success, global_id 4580 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f91a31a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f919b7fe640 10 monclient: dump: 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:44.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.254+0000 7f91a31a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:44.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.260+0000 7f919b7fe640 1 -- 172.21.15.37:0/2705546414 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9194001be0 con 0x7f919c14be90 2024-03-25T18:15:44.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.260+0000 7f919b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:44.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.260+0000 7f919b7fe640 1 -- 172.21.15.37:0/2705546414 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9194002030 con 0x7f919c14be90 2024-03-25T18:15:44.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.260+0000 7f91a31a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:44.261 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.260+0000 7f91a31a8640 4 set_mon_vals no callback set 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.260+0000 7f919b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.260+0000 7f919b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.260+0000 7f919b7fe640 10 monclient: dump: 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:44.262 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:44.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.261+0000 7f91a31a8640 10 monclient: shutdown 2024-03-25T18:15:44.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.261+0000 7f91a31a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.261+0000 7f91a31a8640 1 -- 172.21.15.37:0/2705546414 >> v1:172.21.15.37:6790/0 conn(0x7f919c14be90 legacy=0x7f919c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.261+0000 7f91a31a8640 1 -- 172.21.15.37:0/2705546414 shutdown_connections 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.261+0000 7f91a31a8640 1 -- 172.21.15.37:0/2705546414 >> 172.21.15.37:0/2705546414 conn(0x7f919c0617a0 msgr2=0x7f919c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.261+0000 7f91a31a8640 1 -- 172.21.15.37:0/2705546414 shutdown_connections 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.261+0000 7f91a31a8640 1 -- 172.21.15.37:0/2705546414 wait complete. 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.262+0000 7f91a31a8640 10 monclient: build_initial_monmap 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.262+0000 7f91a31a8640 10 monclient: monmap: 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:44.265 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.262+0000 7f91a31a8640 1 librados: starting msgr at 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.262+0000 7f91a31a8640 1 librados: starting objecter 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.262+0000 7f91a31a8640 1 Processor -- start 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.262+0000 7f91a31a8640 1 -- start start 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.262+0000 7f91a31a8640 1 librados: setting wanted keys 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.262+0000 7f91a31a8640 1 librados: calling monclient init 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.262+0000 7f91a31a8640 10 monclient: init 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.263+0000 7f91a31a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.263+0000 7f91a31a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.263+0000 7f91a31a8640 10 monclient(hunting): picked mon.noname-b con 0x7f919c0b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.263+0000 7f91a31a8640 10 monclient(hunting): picked mon.noname-c con 0x7f919c0b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.263+0000 7f91a31a8640 10 monclient(hunting): picked mon.noname-a con 0x7f919c14be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.263+0000 7f91a31a8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919c13b810 con 0x7f919c14be90 2024-03-25T18:15:44.266 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.263+0000 7f91a31a8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919c135550 con 0x7f919c0b1730 2024-03-25T18:15:44.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.263+0000 7f91a31a8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f919c1356e0 con 0x7f919c0b4d60 2024-03-25T18:15:44.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.263+0000 7f91a31a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:44.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.263+0000 7f91a31a8640 10 monclient(hunting): authenticate will time out at 7299.516110s 2024-03-25T18:15:44.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.267+0000 7f91a0f44640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f919c0b1730 0x7f919c134e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:56202/0 (socket says 172.21.15.37:56202) 2024-03-25T18:15:44.268 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.267+0000 7f91a0f44640 1 -- 172.21.15.37:0/3906797139 learned_addr learned my addr 172.21.15.37:0/3906797139 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:44.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.273+0000 7f91997fa640 10 client.?.objecter ms_handle_connect 0x7f919c0b1730 2024-03-25T18:15:44.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.273+0000 7f91997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:44.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.292+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3781949250 0 0) 0x7f919c135550 con 0x7f919c0b1730 2024-03-25T18:15:44.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.292+0000 7f91997fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:44.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.292+0000 7f91997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:44.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.292+0000 7f91997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:44.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.292+0000 7f91997fa640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:15:44.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.292+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9180004eb0 con 0x7f919c0b1730 2024-03-25T18:15:44.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.292+0000 7f91997fa640 10 client.?.objecter ms_handle_connect 0x7f919c14be90 2024-03-25T18:15:44.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.292+0000 7f91997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:44.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.297+0000 7f91997fa640 10 client.?.objecter ms_handle_connect 0x7f919c0b4d60 2024-03-25T18:15:44.298 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.297+0000 7f91997fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:44.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.301+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4040497561 0 0) 0x7f919c13b810 con 0x7f919c14be90 2024-03-25T18:15:44.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.301+0000 7f91997fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:44.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.301+0000 7f91997fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:44.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.301+0000 7f91997fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:44.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.301+0000 7f91997fa640 10 monclient(hunting): my global_id is 4590 2024-03-25T18:15:44.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.301+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f919c135550 con 0x7f919c14be90 2024-03-25T18:15:44.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.308+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4270383090 0 0) 0x7f9180004eb0 con 0x7f919c0b1730 2024-03-25T18:15:44.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.308+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f919c13b810 con 0x7f919c0b1730 2024-03-25T18:15:44.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.308+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9190004080 con 0x7f919c0b1730 2024-03-25T18:15:44.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.308+0000 7f91997fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:44.309 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.308+0000 7f91997fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.308+0000 7f91997fa640 10 monclient(hunting): dump: 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:44.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.308+0000 7f91997fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2949389354 0 0) 0x7f919c13b810 con 0x7f919c0b1730 2024-03-25T18:15:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 >> v1:172.21.15.37:6790/0 conn(0x7f919c0b4d60 legacy=0x7f919c1d74b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 >> v1:172.21.15.37:6789/0 conn(0x7f919c14be90 legacy=0x7f919c1e2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91997fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91997fa640 1 monclient: found mon.b 2024-03-25T18:15:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91997fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91997fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:44.315 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919c135870 con 0x7f919c0b1730 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91997fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91997fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91997fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91a31a8640 5 monclient: authenticate success, global_id 4585 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91a31a8640 10 monclient: _renew_subs 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91a31a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91a31a8640 1 -- 172.21.15.37:0/3906797139 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f919c1381d0 con 0x7f919c0b1730 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91a31a8640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91a31a8640 10 monclient: _renew_subs 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91a31a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91a31a8640 1 -- 172.21.15.37:0/3906797139 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f919c140560 con 0x7f919c0b1730 2024-03-25T18:15:44.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.313+0000 7f91a31a8640 1 librados: init done 2024-03-25T18:15:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.315+0000 7f91a31a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.315+0000 7f91a31a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.315+0000 7f91a31a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:44.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.315+0000 7f91a31a8640 1 -- 172.21.15.37:0/3906797139 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f919c0627b0 con 0x7f919c0b1730 2024-03-25T18:15:44.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.594+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9190000ed0 con 0x7f919c0b1730 2024-03-25T18:15:44.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.594+0000 7f91997fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:44.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.594+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f9190001220 con 0x7f919c0b1730 2024-03-25T18:15:44.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.594+0000 7f91997fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:44.595 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.594+0000 7f91997fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.594+0000 7f91997fa640 10 monclient: dump: 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:44.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.594+0000 7f919affd640 4 set_mon_vals no callback set 2024-03-25T18:15:44.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.595+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f9190001440 con 0x7f919c0b1730 2024-03-25T18:15:44.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.595+0000 7f919bfff640 10 monclient: get_auth_request con 0x7f918006dd10 auth_method 0 2024-03-25T18:15:44.597 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.595+0000 7f91997fa640 10 client.4585.objecter ms_handle_connect 0x7f918006dd10 2024-03-25T18:15:44.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.600+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f9190054790 con 0x7f919c0b1730 2024-03-25T18:15:44.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.600+0000 7f91997fa640 10 client.4585.objecter ms_dispatch 0x7f919c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:44.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.600+0000 7f91997fa640 3 client.4585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:44.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.600+0000 7f91997fa640 3 client.4585.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:44.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.600+0000 7f91997fa640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-25T18:15:44.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.600+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f91900502a0 con 0x7f919c0b1730 2024-03-25T18:15:44.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.600+0000 7f91997fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:44.601 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.600+0000 7f91997fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:44.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.854+0000 7f91a31a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:44.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.854+0000 7f91a31a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:44.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.854+0000 7f91a31a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:15:44.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:44.854+0000 7f91a31a8640 1 -- 172.21.15.37:0/3906797139 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f919c138360 con 0x7f919c0b1730 2024-03-25T18:15:45.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.007+0000 7f91997fa640 1 -- 172.21.15.37:0/3906797139 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f91900521d0 con 0x7f919c0b1730 2024-03-25T18:15:45.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.007+0000 7f91997fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:45.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.007+0000 7f91997fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:45.009 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:15:45.009 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:45.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.009+0000 7f91a31a8640 10 librados: watch_flush enter 2024-03-25T18:15:45.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.009+0000 7f91a31a8640 10 librados: watch_flush exit 2024-03-25T18:15:45.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.009+0000 7f91a31a8640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-25T18:15:45.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.009+0000 7f91a31a8640 10 client.4585.objecter successfully canceled tick 2024-03-25T18:15:45.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.009+0000 7f91a31a8640 1 -- 172.21.15.37:0/3906797139 >> v1:172.21.15.117:6800/3703993921 conn(0x7f918006dd10 legacy=0x7f91800701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:45.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.009+0000 7f91a31a8640 10 monclient: shutdown 2024-03-25T18:15:45.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.009+0000 7f91a31a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:45.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.009+0000 7f91a31a8640 1 -- 172.21.15.37:0/3906797139 >> v1:172.21.15.117:6789/0 conn(0x7f919c0b1730 legacy=0x7f919c134e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:45.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.009+0000 7f91a31a8640 1 -- 172.21.15.37:0/3906797139 shutdown_connections 2024-03-25T18:15:45.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.009+0000 7f91a31a8640 1 -- 172.21.15.37:0/3906797139 >> 172.21.15.37:0/3906797139 conn(0x7f919c0617a0 msgr2=0x7f919c13bfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:45.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.010+0000 7f91a31a8640 1 -- 172.21.15.37:0/3906797139 shutdown_connections 2024-03-25T18:15:45.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.010+0000 7f91a31a8640 1 -- 172.21.15.37:0/3906797139 wait complete. 2024-03-25T18:15:45.012 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:45.010+0000 7f91a31a8640 1 librados: shutdown 2024-03-25T18:15:45.023 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:50.705 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:08:26.050 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:50.710 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:08:26.056 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:50.741 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:08:26.086 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:50.768 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:08:26.113 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:50.773 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:08:26.086 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:50.774 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:08:26.119 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:50.778 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:08:26.091 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:50.810 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:08:26.123 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:50.816 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:08:26.161 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:51.830 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:51.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.948+0000 7fa0ff1cb640 10 monclient: get_monmap_and_config 2024-03-25T18:15:51.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.948+0000 7fa0ff1cb640 10 monclient: build_initial_monmap 2024-03-25T18:15:51.950 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.948+0000 7fa0ff1cb640 10 monclient: monmap: 2024-03-25T18:15:51.950 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:51.950 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:51.950 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:15:51.949372+0000 2024-03-25T18:15:51.950 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:15:51.949372+0000 2024-03-25T18:15:51.950 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:51.950 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:51.950 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:51.950 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:15:51.950 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:15:51.950 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.949+0000 7fa0ff1cb640 1 Processor -- start 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.949+0000 7fa0ff1cb640 1 -- start start 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.949+0000 7fa0ff1cb640 10 monclient: init 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.950+0000 7fa0ff1cb640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.950+0000 7fa0ff1cb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.950+0000 7fa0ff1cb640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f80b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.950+0000 7fa0ff1cb640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f80b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.950+0000 7fa0ff1cb640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f814be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.950+0000 7fa0ff1cb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80af940 con 0x7fa0f80b4d60 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.950+0000 7fa0ff1cb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80b0a70 con 0x7fa0f814be90 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.950+0000 7fa0ff1cb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f80b0d50 con 0x7fa0f80b1730 2024-03-25T18:15:51.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.950+0000 7fa0ff1cb640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:51.953 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.950+0000 7fa0ff1cb640 10 monclient(hunting): authenticate will time out at 7307.202635s 2024-03-25T18:15:51.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.961+0000 7fa0fd768640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fa0f814be90 0x7fa0f814e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:38228/0 (socket says 172.21.15.37:38228) 2024-03-25T18:15:51.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.961+0000 7fa0fd768640 1 -- 172.21.15.37:0/15077363 learned_addr learned my addr 172.21.15.37:0/15077363 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:51.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.970+0000 7fa0f77fe640 1 -- 172.21.15.37:0/15077363 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3906136738 0 0) 0x7fa0f80b0a70 con 0x7fa0f814be90 2024-03-25T18:15:51.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.970+0000 7fa0f77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:15:51.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.970+0000 7fa0f77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:51.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.970+0000 7fa0f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:51.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.970+0000 7fa0f77fe640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:15:51.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.970+0000 7fa0f77fe640 1 -- 172.21.15.37:0/15077363 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0e8003520 con 0x7fa0f814be90 2024-03-25T18:15:51.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 1 -- 172.21.15.37:0/15077363 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 934651393 0 0) 0x7fa0e8003520 con 0x7fa0f814be90 2024-03-25T18:15:51.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 1 -- 172.21.15.37:0/15077363 >> v1:172.21.15.37:6790/0 conn(0x7fa0f80b1730 legacy=0x7fa0f80b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:51.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 1 -- 172.21.15.37:0/15077363 >> v1:172.21.15.37:6789/0 conn(0x7fa0f80b4d60 legacy=0x7fa0f813f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:51.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:15:51.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 1 monclient: found mon.noname-c 2024-03-25T18:15:51.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:51.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 1 -- 172.21.15.37:0/15077363 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f80b1030 con 0x7fa0f814be90 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 10 monclient: _finish_auth 0 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 1 -- 172.21.15.37:0/15077363 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa0f0001a10 con 0x7fa0f814be90 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0f77fe640 10 monclient: dump: 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:51.984 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:51.985 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:51.985 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:51.985 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:51.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0ff1cb640 5 monclient: authenticate success, global_id 4588 2024-03-25T18:15:51.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.981+0000 7fa0ff1cb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:51.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.993+0000 7fa0f77fe640 1 -- 172.21.15.37:0/15077363 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0f0001b70 con 0x7fa0f814be90 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.993+0000 7fa0f77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.993+0000 7fa0f77fe640 1 -- 172.21.15.37:0/15077363 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa0f0001f80 con 0x7fa0f814be90 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.993+0000 7fa0ff1cb640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.993+0000 7fa0ff1cb640 4 set_mon_vals no callback set 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.993+0000 7fa0f77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.993+0000 7fa0f77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.993+0000 7fa0f77fe640 10 monclient: dump: 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:51.995 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:51.996 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:51.996 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:51.996 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:51.996 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:51.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.994+0000 7fa0ff1cb640 10 monclient: shutdown 2024-03-25T18:15:51.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.994+0000 7fa0ff1cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:51.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.994+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/15077363 >> v1:172.21.15.117:6789/0 conn(0x7fa0f814be90 legacy=0x7fa0f814e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:51.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.994+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/15077363 shutdown_connections 2024-03-25T18:15:51.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.994+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/15077363 >> 172.21.15.37:0/15077363 conn(0x7fa0f80617a0 msgr2=0x7fa0f8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:51.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.994+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/15077363 shutdown_connections 2024-03-25T18:15:51.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.994+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/15077363 wait complete. 2024-03-25T18:15:51.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.995+0000 7fa0ff1cb640 10 monclient: build_initial_monmap 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.995+0000 7fa0ff1cb640 10 monclient: monmap: 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.995+0000 7fa0ff1cb640 1 librados: starting msgr at 2024-03-25T18:15:51.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.995+0000 7fa0ff1cb640 1 librados: starting objecter 2024-03-25T18:15:51.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.996+0000 7fa0ff1cb640 1 Processor -- start 2024-03-25T18:15:51.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.996+0000 7fa0ff1cb640 1 -- start start 2024-03-25T18:15:51.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.996+0000 7fa0ff1cb640 1 librados: setting wanted keys 2024-03-25T18:15:51.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.996+0000 7fa0ff1cb640 1 librados: calling monclient init 2024-03-25T18:15:51.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.996+0000 7fa0ff1cb640 10 monclient: init 2024-03-25T18:15:52.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.996+0000 7fa0ff1cb640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:52.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.996+0000 7fa0ff1cb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:52.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.996+0000 7fa0ff1cb640 10 monclient(hunting): picked mon.noname-a con 0x7fa0f80b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:52.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.996+0000 7fa0ff1cb640 10 monclient(hunting): picked mon.noname-c con 0x7fa0f80b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:52.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.997+0000 7fa0ff1cb640 10 monclient(hunting): picked mon.noname-b con 0x7fa0f814be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:52.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.997+0000 7fa0ff1cb640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f8138640 con 0x7fa0f80b1730 2024-03-25T18:15:52.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.997+0000 7fa0ff1cb640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f8139800 con 0x7fa0f814be90 2024-03-25T18:15:52.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.997+0000 7fa0ff1cb640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa0f8132620 con 0x7fa0f80b4d60 2024-03-25T18:15:52.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.997+0000 7fa0ff1cb640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:52.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:51.997+0000 7fa0ff1cb640 10 monclient(hunting): authenticate will time out at 7307.249394s 2024-03-25T18:15:52.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.007+0000 7fa0fd768640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fa0f814be90 0x7fa0f813b750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:38266/0 (socket says 172.21.15.37:38266) 2024-03-25T18:15:52.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.007+0000 7fa0fd768640 1 -- 172.21.15.37:0/4225036598 learned_addr learned my addr 172.21.15.37:0/4225036598 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:52.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.012+0000 7fa0f57fa640 10 client.?.objecter ms_handle_connect 0x7fa0f814be90 2024-03-25T18:15:52.013 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.012+0000 7fa0f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:52.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.014+0000 7fa0f57fa640 10 client.?.objecter ms_handle_connect 0x7fa0f80b4d60 2024-03-25T18:15:52.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.014+0000 7fa0f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:52.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.014+0000 7fa0f57fa640 10 client.?.objecter ms_handle_connect 0x7fa0f80b1730 2024-03-25T18:15:52.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.014+0000 7fa0f57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:52.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.024+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1180306353 0 0) 0x7fa0f8132620 con 0x7fa0f80b4d60 2024-03-25T18:15:52.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.024+0000 7fa0f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:52.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.024+0000 7fa0f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:52.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.024+0000 7fa0f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:52.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.024+0000 7fa0f57fa640 10 monclient(hunting): my global_id is 4589 2024-03-25T18:15:52.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.024+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0dc004eb0 con 0x7fa0f80b4d60 2024-03-25T18:15:52.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.025+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1636540435 0 0) 0x7fa0f8139800 con 0x7fa0f814be90 2024-03-25T18:15:52.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.025+0000 7fa0f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:52.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.025+0000 7fa0f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:52.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.025+0000 7fa0f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:52.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.025+0000 7fa0f57fa640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:15:52.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.025+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f8132620 con 0x7fa0f814be90 2024-03-25T18:15:52.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.028+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1545291860 0 0) 0x7fa0f8138640 con 0x7fa0f80b1730 2024-03-25T18:15:52.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.028+0000 7fa0f57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:15:52.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.028+0000 7fa0f57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:52.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.028+0000 7fa0f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:52.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.028+0000 7fa0f57fa640 10 monclient(hunting): my global_id is 4596 2024-03-25T18:15:52.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.028+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa0f8139800 con 0x7fa0f80b1730 2024-03-25T18:15:52.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.035+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2490013909 0 0) 0x7fa0dc004eb0 con 0x7fa0f80b4d60 2024-03-25T18:15:52.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.035+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa0f8138640 con 0x7fa0f80b4d60 2024-03-25T18:15:52.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.035+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa0e4001930 con 0x7fa0f80b4d60 2024-03-25T18:15:52.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.035+0000 7fa0f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:52.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.035+0000 7fa0f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:52.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.035+0000 7fa0f57fa640 10 monclient(hunting): dump: 2024-03-25T18:15:52.037 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:52.037 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:52.037 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:52.037 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:52.038 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:52.038 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:52.038 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:52.038 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:52.038 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:52.038 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:52.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.036+0000 7fa0f57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:52.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.039+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 567956608 0 0) 0x7fa0f8138640 con 0x7fa0f80b4d60 2024-03-25T18:15:52.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 >> v1:172.21.15.117:6789/0 conn(0x7fa0f814be90 legacy=0x7fa0f813b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:52.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 >> v1:172.21.15.37:6789/0 conn(0x7fa0f80b1730 legacy=0x7fa0f8131f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:52.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0f57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:15:52.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0f57fa640 1 monclient: found mon.c 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0f57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0f81327b0 con 0x7fa0f80b4d60 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0f57fa640 10 monclient: _finish_auth 0 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0f57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0ff1cb640 5 monclient: authenticate success, global_id 4589 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0ff1cb640 10 monclient: _renew_subs 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0ff1cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/4225036598 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0f8138dd0 con 0x7fa0f80b4d60 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0ff1cb640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0ff1cb640 10 monclient: _renew_subs 2024-03-25T18:15:52.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0ff1cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:52.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/4225036598 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0f8138270 con 0x7fa0f80b4d60 2024-03-25T18:15:52.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.040+0000 7fa0ff1cb640 1 librados: init done 2024-03-25T18:15:52.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.041+0000 7fa0ff1cb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:52.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.041+0000 7fa0ff1cb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:52.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.041+0000 7fa0ff1cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:52.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.041+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/4225036598 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0f80b2270 con 0x7fa0f80b4d60 2024-03-25T18:15:52.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.371+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa0e4003610 con 0x7fa0f80b4d60 2024-03-25T18:15:52.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.371+0000 7fa0f57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:52.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.371+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fa0e4003a20 con 0x7fa0f80b4d60 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.371+0000 7fa0f57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.371+0000 7fa0f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.371+0000 7fa0f57fa640 10 monclient: dump: 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:52.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.371+0000 7fa0f6ffd640 4 set_mon_vals no callback set 2024-03-25T18:15:52.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.372+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fa0e4003c40 con 0x7fa0f80b4d60 2024-03-25T18:15:52.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.373+0000 7fa0fcf67640 10 monclient: get_auth_request con 0x7fa0dc06ddf0 auth_method 0 2024-03-25T18:15:52.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.373+0000 7fa0f57fa640 10 client.4589.objecter ms_handle_connect 0x7fa0dc06ddf0 2024-03-25T18:15:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.377+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fa0e4053610 con 0x7fa0f80b4d60 2024-03-25T18:15:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.377+0000 7fa0f57fa640 10 client.4589.objecter ms_dispatch 0x7fa0f8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.377+0000 7fa0f57fa640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:52.378 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.377+0000 7fa0f57fa640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:52.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.377+0000 7fa0f57fa640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-25T18:15:52.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.377+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa0e404f120 con 0x7fa0f80b4d60 2024-03-25T18:15:52.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.377+0000 7fa0f57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:52.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.377+0000 7fa0f57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:52.634 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.633+0000 7fa0ff1cb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:52.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.633+0000 7fa0ff1cb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:52.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.633+0000 7fa0ff1cb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:15:52.635 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.633+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/4225036598 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa0f81387d0 con 0x7fa0f80b4d60 2024-03-25T18:15:52.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.663+0000 7fa0f57fa640 1 -- 172.21.15.37:0/4225036598 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fa0e4051050 con 0x7fa0f80b4d60 2024-03-25T18:15:52.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.663+0000 7fa0f57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:52.665 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.663+0000 7fa0f57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:52.665 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:15:52.665 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:52.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.665+0000 7fa0ff1cb640 10 librados: watch_flush enter 2024-03-25T18:15:52.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.665+0000 7fa0ff1cb640 10 librados: watch_flush exit 2024-03-25T18:15:52.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.665+0000 7fa0ff1cb640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-25T18:15:52.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.665+0000 7fa0ff1cb640 10 client.4589.objecter successfully canceled tick 2024-03-25T18:15:52.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.666+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/4225036598 >> v1:172.21.15.117:6800/3703993921 conn(0x7fa0dc06ddf0 legacy=0x7fa0dc0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:52.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.666+0000 7fa0ff1cb640 10 monclient: shutdown 2024-03-25T18:15:52.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.666+0000 7fa0ff1cb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:52.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.666+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/4225036598 >> v1:172.21.15.37:6790/0 conn(0x7fa0f80b4d60 legacy=0x7fa0f813b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:52.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.666+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/4225036598 shutdown_connections 2024-03-25T18:15:52.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.666+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/4225036598 >> 172.21.15.37:0/4225036598 conn(0x7fa0f80617a0 msgr2=0x7fa0f813c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:52.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.666+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/4225036598 shutdown_connections 2024-03-25T18:15:52.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.666+0000 7fa0ff1cb640 1 -- 172.21.15.37:0/4225036598 wait complete. 2024-03-25T18:15:52.668 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:52.666+0000 7fa0ff1cb640 1 librados: shutdown 2024-03-25T18:15:52.679 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:59.485 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.600+0000 7f3f38206640 10 monclient: get_monmap_and_config 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.600+0000 7f3f38206640 10 monclient: build_initial_monmap 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.600+0000 7f3f38206640 10 monclient: monmap: 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:15:59.601633+0000 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:15:59.601633+0000 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:15:59.602 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:15:59.603 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:59.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.601+0000 7f3f38206640 1 Processor -- start 2024-03-25T18:15:59.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.601+0000 7f3f38206640 1 -- start start 2024-03-25T18:15:59.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.601+0000 7f3f38206640 10 monclient: init 2024-03-25T18:15:59.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.602+0000 7f3f38206640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:59.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.602+0000 7f3f38206640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:59.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.602+0000 7f3f38206640 10 monclient(hunting): picked mon.noname-c con 0x7f3f3006ab80 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:59.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.602+0000 7f3f38206640 10 monclient(hunting): picked mon.noname-a con 0x7f3f3006af60 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:59.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.602+0000 7f3f38206640 10 monclient(hunting): picked mon.noname-b con 0x7f3f300b00b0 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:59.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.602+0000 7f3f38206640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f300b0500 con 0x7f3f3006af60 2024-03-25T18:15:59.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.602+0000 7f3f38206640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3013ffa0 con 0x7f3f3006ab80 2024-03-25T18:15:59.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.602+0000 7f3f38206640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f300af520 con 0x7f3f300b00b0 2024-03-25T18:15:59.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.602+0000 7f3f38206640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:59.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.602+0000 7f3f38206640 10 monclient(hunting): authenticate will time out at 7314.855010s 2024-03-25T18:15:59.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.609+0000 7f3f357a1640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3f3006af60 0x7f3f3013f890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50638/0 (socket says 172.21.15.37:50638) 2024-03-25T18:15:59.610 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.609+0000 7f3f357a1640 1 -- 172.21.15.37:0/821939963 learned_addr learned my addr 172.21.15.37:0/821939963 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:59.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.625+0000 7f3f34fa0640 1 -- 172.21.15.37:0/821939963 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 247007660 0 0) 0x7f3f300b0500 con 0x7f3f3006af60 2024-03-25T18:15:59.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.625+0000 7f3f34fa0640 10 monclient(hunting): _negotiate 2024-03-25T18:15:59.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.625+0000 7f3f34fa0640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:59.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.625+0000 7f3f34fa0640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:59.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.625+0000 7f3f34fa0640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:15:59.627 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.625+0000 7f3f34fa0640 1 -- 172.21.15.37:0/821939963 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f1c003520 con 0x7f3f3006af60 2024-03-25T18:15:59.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.635+0000 7f3f34fa0640 1 -- 172.21.15.37:0/821939963 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1750364166 0 0) 0x7f3f1c003520 con 0x7f3f3006af60 2024-03-25T18:15:59.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.635+0000 7f3f34fa0640 1 -- 172.21.15.37:0/821939963 >> v1:172.21.15.37:6790/0 conn(0x7f3f300b00b0 legacy=0x7f3f3014dcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:59.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.635+0000 7f3f34fa0640 1 -- 172.21.15.37:0/821939963 >> v1:172.21.15.117:6789/0 conn(0x7f3f3006ab80 legacy=0x7f3f300ad880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:59.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.635+0000 7f3f34fa0640 10 monclient: _finish_hunting 0 2024-03-25T18:15:59.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.635+0000 7f3f34fa0640 1 monclient: found mon.noname-a 2024-03-25T18:15:59.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.635+0000 7f3f34fa0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:59.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.635+0000 7f3f34fa0640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:15:59.637 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.635+0000 7f3f34fa0640 1 -- 172.21.15.37:0/821939963 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f30072890 con 0x7f3f3006af60 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.635+0000 7f3f34fa0640 10 monclient: _finish_auth 0 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.635+0000 7f3f34fa0640 10 monclient: _check_auth_tickets 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.635+0000 7f3f34fa0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.636+0000 7f3f34fa0640 1 -- 172.21.15.37:0/821939963 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3f20001690 con 0x7f3f3006af60 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.636+0000 7f3f34fa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.636+0000 7f3f34fa0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.636+0000 7f3f34fa0640 10 monclient: dump: 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:59.638 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:59.639 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:59.639 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:59.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.636+0000 7f3f38206640 5 monclient: authenticate success, global_id 4599 2024-03-25T18:15:59.639 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.636+0000 7f3f38206640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:59.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.642+0000 7f3f34fa0640 1 -- 172.21.15.37:0/821939963 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f200017f0 con 0x7f3f3006af60 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.642+0000 7f3f34fa0640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.642+0000 7f3f34fa0640 1 -- 172.21.15.37:0/821939963 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3f20001c00 con 0x7f3f3006af60 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.642+0000 7f3f38206640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.642+0000 7f3f38206640 4 set_mon_vals no callback set 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.642+0000 7f3f34fa0640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.642+0000 7f3f34fa0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.642+0000 7f3f34fa0640 10 monclient: dump: 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:59.644 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:59.645 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:59.645 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:59.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.643+0000 7f3f38206640 10 monclient: shutdown 2024-03-25T18:15:59.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.643+0000 7f3f38206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:59.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.643+0000 7f3f38206640 1 -- 172.21.15.37:0/821939963 >> v1:172.21.15.37:6789/0 conn(0x7f3f3006af60 legacy=0x7f3f3013f890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:59.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.643+0000 7f3f38206640 1 -- 172.21.15.37:0/821939963 shutdown_connections 2024-03-25T18:15:59.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.643+0000 7f3f38206640 1 -- 172.21.15.37:0/821939963 >> 172.21.15.37:0/821939963 conn(0x7f3f300b4a80 msgr2=0x7f3f300b4e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.643+0000 7f3f38206640 1 -- 172.21.15.37:0/821939963 shutdown_connections 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.643+0000 7f3f38206640 1 -- 172.21.15.37:0/821939963 wait complete. 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.644+0000 7f3f38206640 10 monclient: build_initial_monmap 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.644+0000 7f3f38206640 10 monclient: monmap: 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:59.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.644+0000 7f3f38206640 1 librados: starting msgr at 2024-03-25T18:15:59.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.644+0000 7f3f38206640 1 librados: starting objecter 2024-03-25T18:15:59.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 1 Processor -- start 2024-03-25T18:15:59.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 1 -- start start 2024-03-25T18:15:59.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 1 librados: setting wanted keys 2024-03-25T18:15:59.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 1 librados: calling monclient init 2024-03-25T18:15:59.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 10 monclient: init 2024-03-25T18:15:59.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:59.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:59.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 10 monclient(hunting): picked mon.noname-b con 0x7f3f3006ab80 addr v1:172.21.15.117:6789/0 2024-03-25T18:15:59.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 10 monclient(hunting): picked mon.noname-c con 0x7f3f300b00b0 addr v1:172.21.15.37:6790/0 2024-03-25T18:15:59.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 10 monclient(hunting): picked mon.noname-a con 0x7f3f30143460 addr v1:172.21.15.37:6789/0 2024-03-25T18:15:59.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f3011e0f0 con 0x7f3f30143460 2024-03-25T18:15:59.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f301daca0 con 0x7f3f3006ab80 2024-03-25T18:15:59.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f301db8d0 con 0x7f3f300b00b0 2024-03-25T18:15:59.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:59.649 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.645+0000 7f3f38206640 10 monclient(hunting): authenticate will time out at 7314.898126s 2024-03-25T18:15:59.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.656+0000 7f3f367a3640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f3f30143460 0x7f3f3013ac20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:50678/0 (socket says 172.21.15.37:50678) 2024-03-25T18:15:59.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.656+0000 7f3f367a3640 1 -- 172.21.15.37:0/3746756600 learned_addr learned my addr 172.21.15.37:0/3746756600 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:15:59.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.656+0000 7f3f267fc640 10 client.?.objecter ms_handle_connect 0x7f3f30143460 2024-03-25T18:15:59.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.656+0000 7f3f267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:59.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.666+0000 7f3f267fc640 10 client.?.objecter ms_handle_connect 0x7f3f300b00b0 2024-03-25T18:15:59.667 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.666+0000 7f3f267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:59.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.668+0000 7f3f267fc640 10 client.?.objecter ms_handle_connect 0x7f3f3006ab80 2024-03-25T18:15:59.669 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.668+0000 7f3f267fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:59.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.674+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 835891128 0 0) 0x7f3f3011e0f0 con 0x7f3f30143460 2024-03-25T18:15:59.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.674+0000 7f3f267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:59.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.674+0000 7f3f267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:59.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.674+0000 7f3f267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:59.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.674+0000 7f3f267fc640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:15:59.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.675+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f14004eb0 con 0x7f3f30143460 2024-03-25T18:15:59.685 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.684+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2133480389 0 0) 0x7f3f301daca0 con 0x7f3f3006ab80 2024-03-25T18:15:59.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.684+0000 7f3f267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:59.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.684+0000 7f3f267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:59.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.684+0000 7f3f267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:59.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.684+0000 7f3f267fc640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:15:59.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.684+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f3011e0f0 con 0x7f3f3006ab80 2024-03-25T18:15:59.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.688+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3696572495 0 0) 0x7f3f301db8d0 con 0x7f3f300b00b0 2024-03-25T18:15:59.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.688+0000 7f3f267fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:59.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.688+0000 7f3f267fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:59.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.688+0000 7f3f267fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:59.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.688+0000 7f3f267fc640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:15:59.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.688+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f301daca0 con 0x7f3f300b00b0 2024-03-25T18:15:59.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.690+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2306764614 0 0) 0x7f3f14004eb0 con 0x7f3f30143460 2024-03-25T18:15:59.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.690+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f301db8d0 con 0x7f3f30143460 2024-03-25T18:15:59.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.690+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3f28001c90 con 0x7f3f30143460 2024-03-25T18:15:59.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.690+0000 7f3f267fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:59.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.690+0000 7f3f267fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:59.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.690+0000 7f3f267fc640 10 monclient(hunting): dump: 2024-03-25T18:15:59.691 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:59.692 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:59.692 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:59.692 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:59.692 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:59.692 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:59.692 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:59.692 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:59.692 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:59.692 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:59.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.690+0000 7f3f267fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:59.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3575696986 0 0) 0x7f3f301db8d0 con 0x7f3f30143460 2024-03-25T18:15:59.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 >> v1:172.21.15.37:6790/0 conn(0x7f3f300b00b0 legacy=0x7f3f301db1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 >> v1:172.21.15.117:6789/0 conn(0x7f3f3006ab80 legacy=0x7f3f30145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f267fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f267fc640 1 monclient: found mon.a 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f267fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f267fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f301dba60 con 0x7f3f30143460 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f267fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f267fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f267fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f38206640 5 monclient: authenticate success, global_id 4602 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f38206640 10 monclient: _renew_subs 2024-03-25T18:15:59.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f38206640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:15:59.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f38206640 1 -- 172.21.15.37:0/3746756600 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f3013b330 con 0x7f3f30143460 2024-03-25T18:15:59.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f38206640 10 client.4602.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:59.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f38206640 10 monclient: _renew_subs 2024-03-25T18:15:59.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f38206640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:15:59.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f38206640 1 -- 172.21.15.37:0/3746756600 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f3011cd00 con 0x7f3f30143460 2024-03-25T18:15:59.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.694+0000 7f3f38206640 1 librados: init done 2024-03-25T18:15:59.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.695+0000 7f3f38206640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:59.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.695+0000 7f3f38206640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:59.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.695+0000 7f3f38206640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:15:59.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.695+0000 7f3f38206640 1 -- 172.21.15.37:0/3746756600 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f301dab00 con 0x7f3f30143460 2024-03-25T18:15:59.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.994+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f28003990 con 0x7f3f30143460 2024-03-25T18:15:59.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.994+0000 7f3f267fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:59.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.994+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f3f28003b40 con 0x7f3f30143460 2024-03-25T18:15:59.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.994+0000 7f3f267fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:59.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.994+0000 7f3f267fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:59.996 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.994+0000 7f3f267fc640 10 monclient: dump: 2024-03-25T18:15:59.996 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:15:59.996 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:15:59.996 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:59.996 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:15:59.996 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:59.997 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:15:59.997 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:15:59.997 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:15:59.997 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:15:59.997 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:15:59.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.994+0000 7f3f27fff640 4 set_mon_vals no callback set 2024-03-25T18:15:59.997 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.995+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f3f28003d60 con 0x7f3f30143460 2024-03-25T18:15:59.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.996+0000 7f3f35fa2640 10 monclient: get_auth_request con 0x7f3f1406ddf0 auth_method 0 2024-03-25T18:15:59.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:15:59.996+0000 7f3f267fc640 10 client.4602.objecter ms_handle_connect 0x7f3f1406ddf0 2024-03-25T18:16:00.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.001+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f3f280547e0 con 0x7f3f30143460 2024-03-25T18:16:00.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.001+0000 7f3f267fc640 10 client.4602.objecter ms_dispatch 0x7f3f30000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:00.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.001+0000 7f3f267fc640 3 client.4602.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:00.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.001+0000 7f3f267fc640 3 client.4602.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:00.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.001+0000 7f3f267fc640 20 client.4602.objecter dump_active .. 0 homeless 2024-03-25T18:16:00.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.001+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f280502f0 con 0x7f3f30143460 2024-03-25T18:16:00.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.001+0000 7f3f267fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:00.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.001+0000 7f3f267fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:00.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.265+0000 7f3f38206640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:00.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.265+0000 7f3f38206640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:00.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.265+0000 7f3f38206640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:16:00.267 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.265+0000 7f3f38206640 1 -- 172.21.15.37:0/3746756600 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f301dba60 con 0x7f3f30143460 2024-03-25T18:16:00.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.307+0000 7f3f267fc640 1 -- 172.21.15.37:0/3746756600 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f3f301dba60 con 0x7f3f30143460 2024-03-25T18:16:00.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.307+0000 7f3f267fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:00.308 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.307+0000 7f3f267fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:00.308 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:16:00.309 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.309+0000 7f3f38206640 10 librados: watch_flush enter 2024-03-25T18:16:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.309+0000 7f3f38206640 10 librados: watch_flush exit 2024-03-25T18:16:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.309+0000 7f3f38206640 20 client.4602.objecter shutdown clearing up homeless session... 2024-03-25T18:16:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.309+0000 7f3f38206640 10 client.4602.objecter successfully canceled tick 2024-03-25T18:16:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.309+0000 7f3f38206640 1 -- 172.21.15.37:0/3746756600 >> v1:172.21.15.117:6800/3703993921 conn(0x7f3f1406ddf0 legacy=0x7f3f140702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.309+0000 7f3f38206640 10 monclient: shutdown 2024-03-25T18:16:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.309+0000 7f3f38206640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:00.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.309+0000 7f3f38206640 1 -- 172.21.15.37:0/3746756600 >> v1:172.21.15.37:6789/0 conn(0x7f3f30143460 legacy=0x7f3f3013ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:00.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.309+0000 7f3f38206640 1 -- 172.21.15.37:0/3746756600 shutdown_connections 2024-03-25T18:16:00.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.310+0000 7f3f38206640 1 -- 172.21.15.37:0/3746756600 >> 172.21.15.37:0/3746756600 conn(0x7f3f300b4a80 msgr2=0x7f3f3006f0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:00.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.310+0000 7f3f38206640 1 -- 172.21.15.37:0/3746756600 shutdown_connections 2024-03-25T18:16:00.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.310+0000 7f3f38206640 1 -- 172.21.15.37:0/3746756600 wait complete. 2024-03-25T18:16:00.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:00.310+0000 7f3f38206640 1 librados: shutdown 2024-03-25T18:16:00.323 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:07.128 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:07.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.244+0000 7f0f8576b640 10 monclient: get_monmap_and_config 2024-03-25T18:16:07.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.244+0000 7f0f8576b640 10 monclient: build_initial_monmap 2024-03-25T18:16:07.246 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.244+0000 7f0f8576b640 10 monclient: monmap: 2024-03-25T18:16:07.246 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:07.246 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:07.247 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:16:07.245831+0000 2024-03-25T18:16:07.247 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:16:07.245831+0000 2024-03-25T18:16:07.247 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:07.247 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:07.247 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:07.247 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:16:07.247 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:16:07.247 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:07.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.245+0000 7f0f8576b640 1 Processor -- start 2024-03-25T18:16:07.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.245+0000 7f0f8576b640 1 -- start start 2024-03-25T18:16:07.248 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.245+0000 7f0f8576b640 10 monclient: init 2024-03-25T18:16:07.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.246+0000 7f0f8576b640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:07.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.246+0000 7f0f8576b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:07.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.246+0000 7f0f8576b640 10 monclient(hunting): picked mon.noname-c con 0x7f0f800a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:07.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.246+0000 7f0f8576b640 10 monclient(hunting): picked mon.noname-b con 0x7f0f8013f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:07.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.246+0000 7f0f8576b640 10 monclient(hunting): picked mon.noname-a con 0x7f0f801430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:07.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.246+0000 7f0f8576b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f80142520 con 0x7f0f801430c0 2024-03-25T18:16:07.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.246+0000 7f0f8576b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f800b28a0 con 0x7f0f800a98f0 2024-03-25T18:16:07.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.246+0000 7f0f8576b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f800b2a30 con 0x7f0f8013f710 2024-03-25T18:16:07.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.246+0000 7f0f8576b640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:07.249 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.246+0000 7f0f8576b640 10 monclient(hunting): authenticate will time out at 7322.498793s 2024-03-25T18:16:07.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.263+0000 7f0f7f7fe640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0f801430c0 0x7f0f8014dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:55522/0 (socket says 172.21.15.37:55522) 2024-03-25T18:16:07.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.263+0000 7f0f7f7fe640 1 -- 172.21.15.37:0/1951955042 learned_addr learned my addr 172.21.15.37:0/1951955042 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:07.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.274+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 722267429 0 0) 0x7f0f80142520 con 0x7f0f801430c0 2024-03-25T18:16:07.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.274+0000 7f0f7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:07.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.274+0000 7f0f7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:07.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.274+0000 7f0f7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:07.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.274+0000 7f0f7e7fc640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:16:07.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.274+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f6c003520 con 0x7f0f801430c0 2024-03-25T18:16:07.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.281+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1951546649 0 0) 0x7f0f800b28a0 con 0x7f0f800a98f0 2024-03-25T18:16:07.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.281+0000 7f0f7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:07.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.281+0000 7f0f7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:07.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.281+0000 7f0f7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:07.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.281+0000 7f0f7e7fc640 10 monclient(hunting): my global_id is 4600 2024-03-25T18:16:07.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.281+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f80142520 con 0x7f0f800a98f0 2024-03-25T18:16:07.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.288+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3137743885 0 0) 0x7f0f800b2a30 con 0x7f0f8013f710 2024-03-25T18:16:07.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.288+0000 7f0f7e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:07.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.288+0000 7f0f7e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:07.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.288+0000 7f0f7e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:07.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.288+0000 7f0f7e7fc640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:16:07.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.288+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f800b28a0 con 0x7f0f8013f710 2024-03-25T18:16:07.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2708499694 0 0) 0x7f0f6c003520 con 0x7f0f801430c0 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 >> v1:172.21.15.37:6790/0 conn(0x7f0f8013f710 legacy=0x7f0f801429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 >> v1:172.21.15.117:6789/0 conn(0x7f0f800a98f0 legacy=0x7f0f800a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 1 monclient: found mon.noname-a 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f800b44c0 con 0x7f0f801430c0 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0f78001a10 con 0x7f0f801430c0 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f8576b640 5 monclient: authenticate success, global_id 4605 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f8576b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:07.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f7e7fc640 10 monclient: dump: 2024-03-25T18:16:07.306 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:07.306 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:07.306 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.306 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.306 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:07.306 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:07.306 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:07.306 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:07.306 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:07.306 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:07.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.303+0000 7f0f8576b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.321+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f78001b70 con 0x7f0f801430c0 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.321+0000 7f0f7e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.321+0000 7f0f7e7fc640 1 -- 172.21.15.37:0/1951955042 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0f78001f80 con 0x7f0f801430c0 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.321+0000 7f0f8576b640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.321+0000 7f0f8576b640 4 set_mon_vals no callback set 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.321+0000 7f0f7e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.321+0000 7f0f7e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.321+0000 7f0f7e7fc640 10 monclient: dump: 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.323 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.324 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:07.324 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:07.324 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:07.324 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:07.324 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:07.324 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.322+0000 7f0f8576b640 10 monclient: shutdown 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.322+0000 7f0f8576b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.322+0000 7f0f8576b640 1 -- 172.21.15.37:0/1951955042 >> v1:172.21.15.37:6789/0 conn(0x7f0f801430c0 legacy=0x7f0f8014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.322+0000 7f0f8576b640 1 -- 172.21.15.37:0/1951955042 shutdown_connections 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.322+0000 7f0f8576b640 1 -- 172.21.15.37:0/1951955042 >> 172.21.15.37:0/1951955042 conn(0x7f0f800b7ea0 msgr2=0x7f0f800b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.322+0000 7f0f8576b640 1 -- 172.21.15.37:0/1951955042 shutdown_connections 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.322+0000 7f0f8576b640 1 -- 172.21.15.37:0/1951955042 wait complete. 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.323+0000 7f0f8576b640 10 monclient: build_initial_monmap 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.323+0000 7f0f8576b640 10 monclient: monmap: 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:16:07.326 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.323+0000 7f0f8576b640 1 librados: starting msgr at 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.323+0000 7f0f8576b640 1 librados: starting objecter 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.323+0000 7f0f8576b640 1 Processor -- start 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.323+0000 7f0f8576b640 1 -- start start 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.323+0000 7f0f8576b640 1 librados: setting wanted keys 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.323+0000 7f0f8576b640 1 librados: calling monclient init 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.323+0000 7f0f8576b640 10 monclient: init 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.324+0000 7f0f8576b640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.324+0000 7f0f8576b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.324+0000 7f0f8576b640 10 monclient(hunting): picked mon.noname-a con 0x7f0f800a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:07.327 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.324+0000 7f0f8576b640 10 monclient(hunting): picked mon.noname-b con 0x7f0f8013f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:07.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.324+0000 7f0f8576b640 10 monclient(hunting): picked mon.noname-c con 0x7f0f801430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:07.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.324+0000 7f0f8576b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f80136a50 con 0x7f0f800a98f0 2024-03-25T18:16:07.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.324+0000 7f0f8576b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f800b79c0 con 0x7f0f8013f710 2024-03-25T18:16:07.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.324+0000 7f0f8576b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f800b7ca0 con 0x7f0f801430c0 2024-03-25T18:16:07.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.324+0000 7f0f8576b640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:07.328 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.324+0000 7f0f8576b640 10 monclient(hunting): authenticate will time out at 7322.576861s 2024-03-25T18:16:07.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.335+0000 7f0f77fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f0f8013f710 0x7f0f80135350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:55524/0 (socket says 172.21.15.37:55524) 2024-03-25T18:16:07.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.335+0000 7f0f77fff640 1 -- 172.21.15.37:0/1904813026 learned_addr learned my addr 172.21.15.37:0/1904813026 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:07.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.336+0000 7f0f777fe640 10 client.?.objecter ms_handle_connect 0x7f0f8013f710 2024-03-25T18:16:07.337 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.336+0000 7f0f777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:07.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.341+0000 7f0f777fe640 10 client.?.objecter ms_handle_connect 0x7f0f801430c0 2024-03-25T18:16:07.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.341+0000 7f0f777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:07.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.341+0000 7f0f777fe640 10 client.?.objecter ms_handle_connect 0x7f0f800a98f0 2024-03-25T18:16:07.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.341+0000 7f0f777fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:07.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.345+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039773777 0 0) 0x7f0f800b7ca0 con 0x7f0f801430c0 2024-03-25T18:16:07.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.345+0000 7f0f777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:07.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.345+0000 7f0f777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:07.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.345+0000 7f0f777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:07.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.345+0000 7f0f777fe640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:16:07.347 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.345+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f60004eb0 con 0x7f0f801430c0 2024-03-25T18:16:07.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.350+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4244153445 0 0) 0x7f0f80136a50 con 0x7f0f800a98f0 2024-03-25T18:16:07.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.350+0000 7f0f777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:07.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.350+0000 7f0f777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:07.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.350+0000 7f0f777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:07.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.350+0000 7f0f777fe640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:16:07.351 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.350+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f800b7ca0 con 0x7f0f800a98f0 2024-03-25T18:16:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.358+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2714190275 0 0) 0x7f0f800b79c0 con 0x7f0f8013f710 2024-03-25T18:16:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.358+0000 7f0f777fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.358+0000 7f0f777fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.358+0000 7f0f777fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.358+0000 7f0f777fe640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:16:07.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.358+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f80136a50 con 0x7f0f8013f710 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.374+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1147325739 0 0) 0x7f0f800b7ca0 con 0x7f0f800a98f0 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.374+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f800b79c0 con 0x7f0f800a98f0 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.374+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0f70001970 con 0x7f0f800a98f0 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.374+0000 7f0f777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.374+0000 7f0f777fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.374+0000 7f0f777fe640 10 monclient(hunting): dump: 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:07.376 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:07.377 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:07.377 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:07.377 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:07.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.374+0000 7f0f777fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:07.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.378+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1588178537 0 0) 0x7f0f60004eb0 con 0x7f0f801430c0 2024-03-25T18:16:07.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.378+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f800b7ca0 con 0x7f0f801430c0 2024-03-25T18:16:07.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.378+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0f78001e80 con 0x7f0f801430c0 2024-03-25T18:16:07.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.378+0000 7f0f777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:07.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.378+0000 7f0f777fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:07.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.378+0000 7f0f777fe640 10 monclient(hunting): dump: 2024-03-25T18:16:07.379 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:07.379 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:07.379 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.379 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.379 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:07.380 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:07.380 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:07.380 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:07.380 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:07.380 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:07.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.382+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3113311237 0 0) 0x7f0f80136a50 con 0x7f0f8013f710 2024-03-25T18:16:07.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.382+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f60004eb0 con 0x7f0f8013f710 2024-03-25T18:16:07.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.382+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0f680019b0 con 0x7f0f8013f710 2024-03-25T18:16:07.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.382+0000 7f0f777fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:07.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.382+0000 7f0f777fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:07.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.382+0000 7f0f777fe640 10 monclient(hunting): dump: 2024-03-25T18:16:07.384 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:07.384 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:07.384 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.384 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.384 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:07.385 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:07.385 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:07.385 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:07.385 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:07.385 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:07.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 981189086 0 0) 0x7f0f800b79c0 con 0x7f0f800a98f0 2024-03-25T18:16:07.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 >> v1:172.21.15.37:6790/0 conn(0x7f0f801430c0 legacy=0x7f0f801e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:07.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 >> v1:172.21.15.117:6789/0 conn(0x7f0f8013f710 legacy=0x7f0f80135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:07.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f777fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:07.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f777fe640 1 monclient: found mon.a 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f777fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f80136ca0 con 0x7f0f800a98f0 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f777fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f777fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f8576b640 5 monclient: authenticate success, global_id 4608 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f8576b640 10 monclient: _renew_subs 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f8576b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f8576b640 1 -- 172.21.15.37:0/1904813026 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f80136f80 con 0x7f0f800a98f0 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.383+0000 7f0f8576b640 10 client.4608.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.384+0000 7f0f8576b640 10 monclient: _renew_subs 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.384+0000 7f0f8576b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.384+0000 7f0f8576b640 1 -- 172.21.15.37:0/1904813026 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f801360e0 con 0x7f0f800a98f0 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.384+0000 7f0f8576b640 1 librados: init done 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.385+0000 7f0f8576b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:07.387 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.385+0000 7f0f8576b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:07.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.385+0000 7f0f8576b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:16:07.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.385+0000 7f0f8576b640 1 -- 172.21.15.37:0/1904813026 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f800627b0 con 0x7f0f800a98f0 2024-03-25T18:16:07.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.706+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f700035f0 con 0x7f0f800a98f0 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.706+0000 7f0f777fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.706+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0f70003900 con 0x7f0f800a98f0 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.706+0000 7f0f777fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.706+0000 7f0f777fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.706+0000 7f0f777fe640 10 monclient: dump: 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:07.708 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:07.709 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:07.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.706+0000 7f0f7dffb640 4 set_mon_vals no callback set 2024-03-25T18:16:07.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.707+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f0f700198d0 con 0x7f0f800a98f0 2024-03-25T18:16:07.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.708+0000 7f0f77fff640 10 monclient: get_auth_request con 0x7f0f6006e0f0 auth_method 0 2024-03-25T18:16:07.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.708+0000 7f0f777fe640 10 client.4608.objecter ms_handle_connect 0x7f0f6006e0f0 2024-03-25T18:16:07.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.712+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f0f70053580 con 0x7f0f800a98f0 2024-03-25T18:16:07.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.712+0000 7f0f777fe640 10 client.4608.objecter ms_dispatch 0x7f0f80000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:07.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.712+0000 7f0f777fe640 3 client.4608.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:07.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.712+0000 7f0f777fe640 3 client.4608.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:07.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.712+0000 7f0f777fe640 20 client.4608.objecter dump_active .. 0 homeless 2024-03-25T18:16:07.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.712+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f7004eec0 con 0x7f0f800a98f0 2024-03-25T18:16:07.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.712+0000 7f0f777fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:07.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.712+0000 7f0f777fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:07.958 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.957+0000 7f0f8576b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:07.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.957+0000 7f0f8576b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:07.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.957+0000 7f0f8576b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:16:07.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.957+0000 7f0f8576b640 1 -- 172.21.15.37:0/1904813026 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f8013e290 con 0x7f0f800a98f0 2024-03-25T18:16:08.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.999+0000 7f0f777fe640 1 -- 172.21.15.37:0/1904813026 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f0f70050fc0 con 0x7f0f800a98f0 2024-03-25T18:16:08.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.999+0000 7f0f777fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:08.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:07.999+0000 7f0f777fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:08.001 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:16:08.001 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:08.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 10 librados: watch_flush enter 2024-03-25T18:16:08.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 10 librados: watch_flush exit 2024-03-25T18:16:08.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 20 client.4608.objecter shutdown clearing up homeless session... 2024-03-25T18:16:08.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 10 client.4608.objecter successfully canceled tick 2024-03-25T18:16:08.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 1 -- 172.21.15.37:0/1904813026 >> v1:172.21.15.117:6800/3703993921 conn(0x7f0f6006e0f0 legacy=0x7f0f600705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:08.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 10 monclient: shutdown 2024-03-25T18:16:08.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:08.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 1 -- 172.21.15.37:0/1904813026 >> v1:172.21.15.37:6789/0 conn(0x7f0f800a98f0 legacy=0x7f0f80137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:08.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 1 -- 172.21.15.37:0/1904813026 shutdown_connections 2024-03-25T18:16:08.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 1 -- 172.21.15.37:0/1904813026 >> 172.21.15.37:0/1904813026 conn(0x7f0f800b7ea0 msgr2=0x7f0f80074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:08.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 1 -- 172.21.15.37:0/1904813026 shutdown_connections 2024-03-25T18:16:08.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 1 -- 172.21.15.37:0/1904813026 wait complete. 2024-03-25T18:16:08.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:08.002+0000 7f0f8576b640 1 librados: shutdown 2024-03-25T18:16:08.018 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:10.755 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:08:46.100 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:10.765 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:08:46.111 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:10.799 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:08:46.145 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:10.822 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:08:46.135 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:10.829 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:08:46.175 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:10.831 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:08:46.144 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:10.847 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:08:46.192 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:10.859 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:08:46.172 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:10.871 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:08:46.216 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:14.824 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:14.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.938+0000 7f2107616640 10 monclient: get_monmap_and_config 2024-03-25T18:16:14.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.938+0000 7f2107616640 10 monclient: build_initial_monmap 2024-03-25T18:16:14.939 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.938+0000 7f2107616640 10 monclient: monmap: 2024-03-25T18:16:14.939 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:14.939 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:14.939 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:16:14.939153+0000 2024-03-25T18:16:14.939 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:16:14.939153+0000 2024-03-25T18:16:14.939 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:14.939 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:14.940 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:14.940 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:16:14.940 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:16:14.940 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:14.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.938+0000 7f2107616640 1 Processor -- start 2024-03-25T18:16:14.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 1 -- start start 2024-03-25T18:16:14.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 10 monclient: init 2024-03-25T18:16:14.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:14.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:14.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 10 monclient(hunting): picked mon.noname-a con 0x7f21000b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:14.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 10 monclient(hunting): picked mon.noname-c con 0x7f21000b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:14.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 10 monclient(hunting): picked mon.noname-b con 0x7f210014be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:14.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21000af940 con 0x7f21000b4d60 2024-03-25T18:16:14.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21000b0a70 con 0x7f21000b1730 2024-03-25T18:16:14.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21000b0d50 con 0x7f210014be90 2024-03-25T18:16:14.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:14.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.939+0000 7f2107616640 10 monclient(hunting): authenticate will time out at 7330.192231s 2024-03-25T18:16:14.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.943+0000 7f2104bb1640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f21000b1730 0x7f21000b1be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:41012/0 (socket says 172.21.15.37:41012) 2024-03-25T18:16:14.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.943+0000 7f2104bb1640 1 -- 172.21.15.37:0/4093841795 learned_addr learned my addr 172.21.15.37:0/4093841795 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:14.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.958+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4082433385 0 0) 0x7f21000af940 con 0x7f21000b4d60 2024-03-25T18:16:14.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.958+0000 7f20f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:14.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.958+0000 7f20f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:14.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.958+0000 7f20f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:14.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.958+0000 7f20f7fff640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:16:14.959 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.958+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20ec003520 con 0x7f21000b4d60 2024-03-25T18:16:14.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.959+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401199909 0 0) 0x7f21000b0a70 con 0x7f21000b1730 2024-03-25T18:16:14.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.959+0000 7f20f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:14.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.959+0000 7f20f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:14.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.959+0000 7f20f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:14.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.959+0000 7f20f7fff640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:16:14.961 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.959+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21000af940 con 0x7f21000b1730 2024-03-25T18:16:14.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.966+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040852436 0 0) 0x7f21000b0d50 con 0x7f210014be90 2024-03-25T18:16:14.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.966+0000 7f20f7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:14.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.966+0000 7f20f7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:14.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.966+0000 7f20f7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:14.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.966+0000 7f20f7fff640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:16:14.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.966+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21000b0a70 con 0x7f210014be90 2024-03-25T18:16:14.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2039522444 0 0) 0x7f20ec003520 con 0x7f21000b4d60 2024-03-25T18:16:14.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 >> v1:172.21.15.37:6790/0 conn(0x7f210014be90 legacy=0x7f210014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:14.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 >> v1:172.21.15.117:6789/0 conn(0x7f21000b1730 legacy=0x7f21000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:14.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:14.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 1 monclient: found mon.noname-a 2024-03-25T18:16:14.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:14.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:16:14.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21000b1030 con 0x7f21000b4d60 2024-03-25T18:16:14.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:14.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f20fc001690 con 0x7f21000b4d60 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f2107616640 5 monclient: authenticate success, global_id 4611 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f2107616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f20f7fff640 10 monclient: dump: 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:14.971 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:14.972 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.968+0000 7f2107616640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:14.974 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.973+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20fc0017f0 con 0x7f21000b4d60 2024-03-25T18:16:14.974 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.973+0000 7f20f7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:14.974 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.973+0000 7f20f7fff640 1 -- 172.21.15.37:0/4093841795 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f20fc001c00 con 0x7f21000b4d60 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.973+0000 7f20f7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.973+0000 7f20f7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.973+0000 7f20f7fff640 10 monclient: dump: 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.973+0000 7f2107616640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:14.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.973+0000 7f2107616640 4 set_mon_vals no callback set 2024-03-25T18:16:14.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.974+0000 7f2107616640 10 monclient: shutdown 2024-03-25T18:16:14.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.974+0000 7f2107616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:14.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.974+0000 7f2107616640 1 -- 172.21.15.37:0/4093841795 >> v1:172.21.15.37:6789/0 conn(0x7f21000b4d60 legacy=0x7f210013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:14.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.974+0000 7f2107616640 1 -- 172.21.15.37:0/4093841795 shutdown_connections 2024-03-25T18:16:14.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.974+0000 7f2107616640 1 -- 172.21.15.37:0/4093841795 >> 172.21.15.37:0/4093841795 conn(0x7f21000617a0 msgr2=0x7f2100061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:14.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.974+0000 7f2107616640 1 -- 172.21.15.37:0/4093841795 shutdown_connections 2024-03-25T18:16:14.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.974+0000 7f2107616640 1 -- 172.21.15.37:0/4093841795 wait complete. 2024-03-25T18:16:14.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.975+0000 7f2107616640 10 monclient: build_initial_monmap 2024-03-25T18:16:14.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.975+0000 7f2107616640 10 monclient: monmap: 2024-03-25T18:16:14.977 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.976+0000 7f2107616640 1 librados: starting msgr at 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.976+0000 7f2107616640 1 librados: starting objecter 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.976+0000 7f2107616640 1 Processor -- start 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.976+0000 7f2107616640 1 -- start start 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.976+0000 7f2107616640 1 librados: setting wanted keys 2024-03-25T18:16:14.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.976+0000 7f2107616640 1 librados: calling monclient init 2024-03-25T18:16:14.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.976+0000 7f2107616640 10 monclient: init 2024-03-25T18:16:14.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.976+0000 7f2107616640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:14.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.977+0000 7f2107616640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:14.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.977+0000 7f2107616640 10 monclient(hunting): picked mon.noname-c con 0x7f21000b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:14.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.977+0000 7f2107616640 10 monclient(hunting): picked mon.noname-b con 0x7f21000b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:14.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.977+0000 7f2107616640 10 monclient(hunting): picked mon.noname-a con 0x7f210014be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:14.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.977+0000 7f2107616640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f210013aba0 con 0x7f210014be90 2024-03-25T18:16:14.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.977+0000 7f2107616640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21001414a0 con 0x7f21000b4d60 2024-03-25T18:16:14.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.977+0000 7f2107616640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21001dfc90 con 0x7f21000b1730 2024-03-25T18:16:14.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.977+0000 7f2107616640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:14.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.977+0000 7f2107616640 10 monclient(hunting): authenticate will time out at 7330.229597s 2024-03-25T18:16:14.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.988+0000 7f2104bb1640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f21000b4d60 0x7f2100142b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:41032/0 (socket says 172.21.15.37:41032) 2024-03-25T18:16:14.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.988+0000 7f2104bb1640 1 -- 172.21.15.37:0/2560634455 learned_addr learned my addr 172.21.15.37:0/2560634455 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:14.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.989+0000 7f20f5ffb640 10 client.?.objecter ms_handle_connect 0x7f21000b4d60 2024-03-25T18:16:14.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.989+0000 7f20f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:14.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.990+0000 7f20f5ffb640 10 client.?.objecter ms_handle_connect 0x7f210014be90 2024-03-25T18:16:14.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.990+0000 7f20f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:14.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.992+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 336193149 0 0) 0x7f21001414a0 con 0x7f21000b4d60 2024-03-25T18:16:14.993 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.992+0000 7f20f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:14.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.992+0000 7f20f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:14.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.992+0000 7f20f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:14.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.992+0000 7f20f5ffb640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:16:14.994 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.992+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e4004eb0 con 0x7f21000b4d60 2024-03-25T18:16:15.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.999+0000 7f20f5ffb640 10 client.?.objecter ms_handle_connect 0x7f21000b1730 2024-03-25T18:16:15.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:14.999+0000 7f20f5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:15.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.001+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3944674652 0 0) 0x7f20e4004eb0 con 0x7f21000b4d60 2024-03-25T18:16:15.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.001+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f21001414a0 con 0x7f21000b4d60 2024-03-25T18:16:15.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.001+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f20f0001970 con 0x7f21000b4d60 2024-03-25T18:16:15.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.001+0000 7f20f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.001+0000 7f20f5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.001+0000 7f20f5ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:15.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.001+0000 7f20f5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:15.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.007+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2356161918 0 0) 0x7f21001dfc90 con 0x7f21000b1730 2024-03-25T18:16:15.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.007+0000 7f20f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:15.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.007+0000 7f20f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:15.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.007+0000 7f20f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:15.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.007+0000 7f20f5ffb640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:16:15.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.007+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20e4004eb0 con 0x7f21000b1730 2024-03-25T18:16:15.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.009+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 462035328 0 0) 0x7f210013aba0 con 0x7f210014be90 2024-03-25T18:16:15.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.009+0000 7f20f5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:15.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.009+0000 7f20f5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:15.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.009+0000 7f20f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:15.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.009+0000 7f20f5ffb640 10 monclient(hunting): my global_id is 4614 2024-03-25T18:16:15.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.009+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f21001dfc90 con 0x7f210014be90 2024-03-25T18:16:15.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.023+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 95566580 0 0) 0x7f20e4004eb0 con 0x7f21000b1730 2024-03-25T18:16:15.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.023+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f210013aba0 con 0x7f21000b1730 2024-03-25T18:16:15.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.023+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f20fc001b00 con 0x7f21000b1730 2024-03-25T18:16:15.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.023+0000 7f20f5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:15.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.023+0000 7f20f5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:15.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.023+0000 7f20f5ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:15.025 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:15.025 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:15.025 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:15.025 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:15.025 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:15.025 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:15.025 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:15.025 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:15.025 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:15.025 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1538048592 0 0) 0x7f21001414a0 con 0x7f21000b4d60 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 >> v1:172.21.15.37:6790/0 conn(0x7f21000b1730 legacy=0x7f21001df580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 >> v1:172.21.15.37:6789/0 conn(0x7f210014be90 legacy=0x7f21001394e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f20f5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f20f5ffb640 1 monclient: found mon.b 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f20f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f20f5ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21001dfe20 con 0x7f21000b4d60 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f20f5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f20f5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f20f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:15.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f2107616640 5 monclient: authenticate success, global_id 4609 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f2107616640 10 monclient: _renew_subs 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f2107616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f2107616640 1 -- 172.21.15.37:0/2560634455 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2100145160 con 0x7f21000b4d60 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f2107616640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f2107616640 10 monclient: _renew_subs 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f2107616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f2107616640 1 -- 172.21.15.37:0/2560634455 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2100143280 con 0x7f21000b4d60 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.024+0000 7f2107616640 1 librados: init done 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.025+0000 7f2107616640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.025+0000 7f2107616640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.025+0000 7f2107616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:15.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.025+0000 7f2107616640 1 -- 172.21.15.37:0/2560634455 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21001dffb0 con 0x7f21000b4d60 2024-03-25T18:16:15.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.299+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f20f00047e0 con 0x7f21000b4d60 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.299+0000 7f20f5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.299+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f20f0004af0 con 0x7f21000b4d60 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.299+0000 7f20f5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.299+0000 7f20f5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.299+0000 7f20f5ffb640 10 monclient: dump: 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:15.301 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:15.302 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:15.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.299+0000 7f20f77fe640 4 set_mon_vals no callback set 2024-03-25T18:16:15.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.300+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f20f001ab00 con 0x7f21000b4d60 2024-03-25T18:16:15.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.301+0000 7f21053b2640 10 monclient: get_auth_request con 0x7f20e406df50 auth_method 0 2024-03-25T18:16:15.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.301+0000 7f20f5ffb640 10 client.4609.objecter ms_handle_connect 0x7f20e406df50 2024-03-25T18:16:15.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.305+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f20f0054850 con 0x7f21000b4d60 2024-03-25T18:16:15.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.305+0000 7f20f5ffb640 10 client.4609.objecter ms_dispatch 0x7f2100000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:15.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.305+0000 7f20f5ffb640 3 client.4609.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:15.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.305+0000 7f20f5ffb640 3 client.4609.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:15.306 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.305+0000 7f20f5ffb640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-25T18:16:15.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.305+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20f0050190 con 0x7f21000b4d60 2024-03-25T18:16:15.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.305+0000 7f20f5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:15.307 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.305+0000 7f20f5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:15.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.552+0000 7f2107616640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:15.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.552+0000 7f2107616640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:15.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.552+0000 7f2107616640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:15.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.552+0000 7f2107616640 1 -- 172.21.15.37:0/2560634455 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f21000627b0 con 0x7f21000b4d60 2024-03-25T18:16:15.574 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.573+0000 7f20f5ffb640 1 -- 172.21.15.37:0/2560634455 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f20f0052290 con 0x7f21000b4d60 2024-03-25T18:16:15.574 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.573+0000 7f20f5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:15.574 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.573+0000 7f20f5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:15.574 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:16:15.575 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:15.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 10 librados: watch_flush enter 2024-03-25T18:16:15.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 10 librados: watch_flush exit 2024-03-25T18:16:15.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-25T18:16:15.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 10 client.4609.objecter successfully canceled tick 2024-03-25T18:16:15.577 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 1 -- 172.21.15.37:0/2560634455 >> v1:172.21.15.117:6800/3703993921 conn(0x7f20e406df50 legacy=0x7f20e4070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:15.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 10 monclient: shutdown 2024-03-25T18:16:15.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:15.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 1 -- 172.21.15.37:0/2560634455 >> v1:172.21.15.117:6789/0 conn(0x7f21000b4d60 legacy=0x7f2100142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:15.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 1 -- 172.21.15.37:0/2560634455 shutdown_connections 2024-03-25T18:16:15.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 1 -- 172.21.15.37:0/2560634455 >> 172.21.15.37:0/2560634455 conn(0x7f21000617a0 msgr2=0x7f210013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:15.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 1 -- 172.21.15.37:0/2560634455 shutdown_connections 2024-03-25T18:16:15.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.575+0000 7f2107616640 1 -- 172.21.15.37:0/2560634455 wait complete. 2024-03-25T18:16:15.578 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:15.576+0000 7f2107616640 1 librados: shutdown 2024-03-25T18:16:15.589 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:22.396 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.516+0000 7f7c7cb4b640 10 monclient: get_monmap_and_config 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.516+0000 7f7c7cb4b640 10 monclient: build_initial_monmap 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.516+0000 7f7c7cb4b640 10 monclient: monmap: 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:16:22.516979+0000 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:16:22.516979+0000 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:16:22.517 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:16:22.518 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:22.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.516+0000 7f7c7cb4b640 1 Processor -- start 2024-03-25T18:16:22.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.516+0000 7f7c7cb4b640 1 -- start start 2024-03-25T18:16:22.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.516+0000 7f7c7cb4b640 10 monclient: init 2024-03-25T18:16:22.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.517+0000 7f7c7cb4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:22.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.517+0000 7f7c7cb4b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:22.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.517+0000 7f7c7cb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f7c780a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:22.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.517+0000 7f7c7cb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f7c7813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:22.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.517+0000 7f7c7cb4b640 10 monclient(hunting): picked mon.noname-c con 0x7f7c781430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:22.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.517+0000 7f7c7cb4b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c78142520 con 0x7f7c7813f710 2024-03-25T18:16:22.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.517+0000 7f7c7cb4b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c780b28a0 con 0x7f7c781430c0 2024-03-25T18:16:22.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.517+0000 7f7c7cb4b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c780b2a30 con 0x7f7c780a98f0 2024-03-25T18:16:22.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.517+0000 7f7c7cb4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:22.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.517+0000 7f7c7cb4b640 10 monclient(hunting): authenticate will time out at 7337.769709s 2024-03-25T18:16:22.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.518+0000 7f7c6dd9b640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f7c7813f710 0x7f7c781429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:48724/0 (socket says 172.21.15.37:48724) 2024-03-25T18:16:22.520 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.518+0000 7f7c6dd9b640 1 -- 172.21.15.37:0/3465959618 learned_addr learned my addr 172.21.15.37:0/3465959618 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:22.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.528+0000 7f7c75d9b640 1 -- 172.21.15.37:0/3465959618 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2047641822 0 0) 0x7f7c78142520 con 0x7f7c7813f710 2024-03-25T18:16:22.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.528+0000 7f7c75d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:22.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.528+0000 7f7c75d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:22.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.528+0000 7f7c75d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:22.529 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.528+0000 7f7c75d9b640 10 monclient(hunting): my global_id is 4617 2024-03-25T18:16:22.530 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.528+0000 7f7c75d9b640 1 -- 172.21.15.37:0/3465959618 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c64003520 con 0x7f7c7813f710 2024-03-25T18:16:22.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 1 -- 172.21.15.37:0/3465959618 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3437324168 0 0) 0x7f7c64003520 con 0x7f7c7813f710 2024-03-25T18:16:22.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 1 -- 172.21.15.37:0/3465959618 >> v1:172.21.15.37:6790/0 conn(0x7f7c780a98f0 legacy=0x7f7c780a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:22.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 1 -- 172.21.15.37:0/3465959618 >> v1:172.21.15.117:6789/0 conn(0x7f7c781430c0 legacy=0x7f7c7814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:22.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:16:22.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 1 monclient: found mon.noname-a 2024-03-25T18:16:22.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:22.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:16:22.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 1 -- 172.21.15.37:0/3465959618 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c780b44c0 con 0x7f7c7813f710 2024-03-25T18:16:22.535 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 10 monclient: _finish_auth 0 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 1 -- 172.21.15.37:0/3465959618 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7c60001690 con 0x7f7c7813f710 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c75d9b640 10 monclient: dump: 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:22.536 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:22.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c7cb4b640 5 monclient: authenticate success, global_id 4617 2024-03-25T18:16:22.537 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.533+0000 7f7c7cb4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:22.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c75d9b640 1 -- 172.21.15.37:0/3465959618 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c60001870 con 0x7f7c7813f710 2024-03-25T18:16:22.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c75d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:22.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c75d9b640 1 -- 172.21.15.37:0/3465959618 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7c60001cc0 con 0x7f7c7813f710 2024-03-25T18:16:22.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c75d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:22.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c75d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:22.542 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c75d9b640 10 monclient: dump: 2024-03-25T18:16:22.542 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:22.542 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:22.543 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:22.543 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:22.543 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:22.543 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:22.543 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:22.543 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:22.543 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:22.543 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c7cb4b640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:22.543 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c7cb4b640 4 set_mon_vals no callback set 2024-03-25T18:16:22.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c7cb4b640 10 monclient: shutdown 2024-03-25T18:16:22.544 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c7cb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/3465959618 >> v1:172.21.15.37:6789/0 conn(0x7f7c7813f710 legacy=0x7f7c781429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/3465959618 shutdown_connections 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.541+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/3465959618 >> 172.21.15.37:0/3465959618 conn(0x7f7c780b7ea0 msgr2=0x7f7c780b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.542+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/3465959618 shutdown_connections 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.542+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/3465959618 wait complete. 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.542+0000 7f7c7cb4b640 10 monclient: build_initial_monmap 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.542+0000 7f7c7cb4b640 10 monclient: monmap: 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:16:22.545 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:22.546 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:16:22.546 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:16:22.546 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:22.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.543+0000 7f7c7cb4b640 1 librados: starting msgr at 2024-03-25T18:16:22.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.543+0000 7f7c7cb4b640 1 librados: starting objecter 2024-03-25T18:16:22.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.543+0000 7f7c7cb4b640 1 Processor -- start 2024-03-25T18:16:22.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.543+0000 7f7c7cb4b640 1 -- start start 2024-03-25T18:16:22.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.543+0000 7f7c7cb4b640 1 librados: setting wanted keys 2024-03-25T18:16:22.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.543+0000 7f7c7cb4b640 1 librados: calling monclient init 2024-03-25T18:16:22.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.543+0000 7f7c7cb4b640 10 monclient: init 2024-03-25T18:16:22.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.544+0000 7f7c7cb4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:22.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.544+0000 7f7c7cb4b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:22.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.544+0000 7f7c7cb4b640 10 monclient(hunting): picked mon.noname-a con 0x7f7c780a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:22.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.544+0000 7f7c7cb4b640 10 monclient(hunting): picked mon.noname-b con 0x7f7c7813f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:22.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.544+0000 7f7c7cb4b640 10 monclient(hunting): picked mon.noname-c con 0x7f7c781430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:22.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.544+0000 7f7c7cb4b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c781d7ca0 con 0x7f7c780a98f0 2024-03-25T18:16:22.547 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.544+0000 7f7c7cb4b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c78137190 con 0x7f7c7813f710 2024-03-25T18:16:22.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.544+0000 7f7c7cb4b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c78137320 con 0x7f7c781430c0 2024-03-25T18:16:22.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.544+0000 7f7c7cb4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:22.548 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.544+0000 7f7c7cb4b640 10 monclient(hunting): authenticate will time out at 7337.796794s 2024-03-25T18:16:22.552 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.551+0000 7f7c6dd9b640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f7c7813f710 0x7f7c78136a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:45902/0 (socket says 172.21.15.37:45902) 2024-03-25T18:16:22.553 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.551+0000 7f7c6dd9b640 1 -- 172.21.15.37:0/1828459919 learned_addr learned my addr 172.21.15.37:0/1828459919 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:22.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.558+0000 7f7c6f7fe640 10 client.?.objecter ms_handle_connect 0x7f7c7813f710 2024-03-25T18:16:22.559 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.558+0000 7f7c6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:22.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.560+0000 7f7c6f7fe640 10 client.?.objecter ms_handle_connect 0x7f7c781430c0 2024-03-25T18:16:22.561 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.560+0000 7f7c6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:22.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.564+0000 7f7c6f7fe640 10 client.?.objecter ms_handle_connect 0x7f7c780a98f0 2024-03-25T18:16:22.565 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.564+0000 7f7c6f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:22.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.568+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007761513 0 0) 0x7f7c78137320 con 0x7f7c781430c0 2024-03-25T18:16:22.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.568+0000 7f7c6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:22.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.568+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:22.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.568+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:22.569 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.568+0000 7f7c6f7fe640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:16:22.570 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.568+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c58004eb0 con 0x7f7c781430c0 2024-03-25T18:16:22.574 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1801291669 0 0) 0x7f7c781d7ca0 con 0x7f7c780a98f0 2024-03-25T18:16:22.574 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:22.574 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:22.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:22.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 10 monclient(hunting): my global_id is 4620 2024-03-25T18:16:22.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c78137320 con 0x7f7c780a98f0 2024-03-25T18:16:22.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4103194182 0 0) 0x7f7c78137190 con 0x7f7c7813f710 2024-03-25T18:16:22.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:22.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:22.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:22.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:16:22.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.573+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c781d7ca0 con 0x7f7c7813f710 2024-03-25T18:16:22.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.579+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1838231253 0 0) 0x7f7c58004eb0 con 0x7f7c781430c0 2024-03-25T18:16:22.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.579+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c78137190 con 0x7f7c781430c0 2024-03-25T18:16:22.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.579+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7c70001b90 con 0x7f7c781430c0 2024-03-25T18:16:22.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.579+0000 7f7c6f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:22.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.579+0000 7f7c6f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:22.580 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.579+0000 7f7c6f7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:22.580 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:22.581 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:22.581 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:22.581 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:22.581 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:22.581 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:22.581 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:22.581 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:22.581 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:22.581 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:22.581 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.579+0000 7f7c6f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:22.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4107282764 0 0) 0x7f7c78137190 con 0x7f7c781430c0 2024-03-25T18:16:22.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 >> v1:172.21.15.117:6789/0 conn(0x7f7c7813f710 legacy=0x7f7c78136a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:22.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 >> v1:172.21.15.37:6789/0 conn(0x7f7c780a98f0 legacy=0x7f7c781d7590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:22.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c6f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:22.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c6f7fe640 1 monclient: found mon.c 2024-03-25T18:16:22.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:22.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c6f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:22.587 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c78133780 con 0x7f7c781430c0 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c6f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c6f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c7cb4b640 5 monclient: authenticate success, global_id 4613 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c7cb4b640 10 monclient: _renew_subs 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c7cb4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/1828459919 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c78133970 con 0x7f7c781430c0 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c7cb4b640 10 client.4613.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c7cb4b640 10 monclient: _renew_subs 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c7cb4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/1828459919 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c78137d40 con 0x7f7c781430c0 2024-03-25T18:16:22.588 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.585+0000 7f7c7cb4b640 1 librados: init done 2024-03-25T18:16:22.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.586+0000 7f7c7cb4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:22.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.586+0000 7f7c7cb4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:22.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.586+0000 7f7c7cb4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:22.589 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.586+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/1828459919 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c78139200 con 0x7f7c781430c0 2024-03-25T18:16:22.908 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.907+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c70002080 con 0x7f7c781430c0 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.907+0000 7f7c6f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.907+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f7c70003b30 con 0x7f7c781430c0 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.907+0000 7f7c6f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.907+0000 7f7c6f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.907+0000 7f7c6f7fe640 10 monclient: dump: 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:22.909 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:22.910 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:22.910 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:22.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.907+0000 7f7c7559a640 4 set_mon_vals no callback set 2024-03-25T18:16:22.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.908+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f7c70003d50 con 0x7f7c781430c0 2024-03-25T18:16:22.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.908+0000 7f7c7659c640 10 monclient: get_auth_request con 0x7f7c5806ddf0 auth_method 0 2024-03-25T18:16:22.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.909+0000 7f7c6f7fe640 10 client.4613.objecter ms_handle_connect 0x7f7c5806ddf0 2024-03-25T18:16:22.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.915+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f7c70053880 con 0x7f7c781430c0 2024-03-25T18:16:22.916 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.915+0000 7f7c6f7fe640 10 client.4613.objecter ms_dispatch 0x7f7c78000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:22.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.915+0000 7f7c6f7fe640 3 client.4613.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:22.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.915+0000 7f7c6f7fe640 3 client.4613.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:22.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.915+0000 7f7c6f7fe640 20 client.4613.objecter dump_active .. 0 homeless 2024-03-25T18:16:22.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.915+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c7004f390 con 0x7f7c781430c0 2024-03-25T18:16:22.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.915+0000 7f7c6f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:22.917 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:22.915+0000 7f7c6f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:23.154 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.153+0000 7f7c7cb4b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:23.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.153+0000 7f7c7cb4b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:23.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.153+0000 7f7c7cb4b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:23.155 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.153+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/1828459919 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c780627b0 con 0x7f7c781430c0 2024-03-25T18:16:23.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.184+0000 7f7c6f7fe640 1 -- 172.21.15.37:0/1828459919 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f7c700512c0 con 0x7f7c781430c0 2024-03-25T18:16:23.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.185+0000 7f7c6f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:23.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.185+0000 7f7c6f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:23.186 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:16:23.186 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:23.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.187+0000 7f7c7cb4b640 10 librados: watch_flush enter 2024-03-25T18:16:23.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 10 librados: watch_flush exit 2024-03-25T18:16:23.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 20 client.4613.objecter shutdown clearing up homeless session... 2024-03-25T18:16:23.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 10 client.4613.objecter successfully canceled tick 2024-03-25T18:16:23.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/1828459919 >> v1:172.21.15.117:6800/3703993921 conn(0x7f7c5806ddf0 legacy=0x7f7c580702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:23.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 10 monclient: shutdown 2024-03-25T18:16:23.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:23.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/1828459919 >> v1:172.21.15.37:6790/0 conn(0x7f7c781430c0 legacy=0x7f7c78133070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:23.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/1828459919 shutdown_connections 2024-03-25T18:16:23.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/1828459919 >> 172.21.15.37:0/1828459919 conn(0x7f7c780b7ea0 msgr2=0x7f7c78074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:23.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/1828459919 shutdown_connections 2024-03-25T18:16:23.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 1 -- 172.21.15.37:0/1828459919 wait complete. 2024-03-25T18:16:23.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:23.188+0000 7f7c7cb4b640 1 librados: shutdown 2024-03-25T18:16:23.201 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:30.008 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:30.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.124+0000 7fdddb996640 10 monclient: get_monmap_and_config 2024-03-25T18:16:30.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.124+0000 7fdddb996640 10 monclient: build_initial_monmap 2024-03-25T18:16:30.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.124+0000 7fdddb996640 10 monclient: monmap: 2024-03-25T18:16:30.126 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:30.126 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:30.126 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:16:30.125831+0000 2024-03-25T18:16:30.126 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:16:30.125831+0000 2024-03-25T18:16:30.126 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:30.127 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:30.127 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:30.127 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:16:30.127 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:16:30.127 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:30.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.125+0000 7fdddb996640 1 Processor -- start 2024-03-25T18:16:30.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.126+0000 7fdddb996640 1 -- start start 2024-03-25T18:16:30.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.126+0000 7fdddb996640 10 monclient: init 2024-03-25T18:16:30.128 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.126+0000 7fdddb996640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:30.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.127+0000 7fdddb996640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:30.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.127+0000 7fdddb996640 10 monclient(hunting): picked mon.noname-c con 0x7fddd40a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:30.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.127+0000 7fdddb996640 10 monclient(hunting): picked mon.noname-b con 0x7fddd413f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:30.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.127+0000 7fdddb996640 10 monclient(hunting): picked mon.noname-a con 0x7fddd41430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:30.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.127+0000 7fdddb996640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd4142520 con 0x7fddd41430c0 2024-03-25T18:16:30.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.127+0000 7fdddb996640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd40b28a0 con 0x7fddd40a98f0 2024-03-25T18:16:30.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.127+0000 7fdddb996640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd40b2a30 con 0x7fddd413f710 2024-03-25T18:16:30.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.127+0000 7fdddb996640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:30.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.127+0000 7fdddb996640 10 monclient(hunting): authenticate will time out at 7345.379543s 2024-03-25T18:16:30.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.132+0000 7fddd9732640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fddd40a98f0 0x7fddd40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:51056/0 (socket says 172.21.15.37:51056) 2024-03-25T18:16:30.133 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.132+0000 7fddd9732640 1 -- 172.21.15.37:0/765312486 learned_addr learned my addr 172.21.15.37:0/765312486 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:30.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.142+0000 7fddcbfff640 1 -- 172.21.15.37:0/765312486 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1014949498 0 0) 0x7fddd40b28a0 con 0x7fddd40a98f0 2024-03-25T18:16:30.143 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.142+0000 7fddcbfff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:30.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.142+0000 7fddcbfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:30.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.142+0000 7fddcbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:30.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.142+0000 7fddcbfff640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:16:30.144 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.142+0000 7fddcbfff640 1 -- 172.21.15.37:0/765312486 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddc0003520 con 0x7fddd40a98f0 2024-03-25T18:16:30.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 1 -- 172.21.15.37:0/765312486 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1423763762 0 0) 0x7fddc0003520 con 0x7fddd40a98f0 2024-03-25T18:16:30.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 1 -- 172.21.15.37:0/765312486 >> v1:172.21.15.37:6790/0 conn(0x7fddd413f710 legacy=0x7fddd41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 1 -- 172.21.15.37:0/765312486 >> v1:172.21.15.37:6789/0 conn(0x7fddd41430c0 legacy=0x7fddd414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 1 monclient: found mon.noname-c 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 1 -- 172.21.15.37:0/765312486 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd40b44c0 con 0x7fddd40a98f0 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 10 monclient: _finish_auth 0 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fdddb996640 5 monclient: authenticate success, global_id 4618 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 1 -- 172.21.15.37:0/765312486 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fddd0001690 con 0x7fddd40a98f0 2024-03-25T18:16:30.157 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fdddb996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fddcbfff640 10 monclient: dump: 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:30.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.155+0000 7fdddb996640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:30.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.164+0000 7fddcbfff640 1 -- 172.21.15.37:0/765312486 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddd00018b0 con 0x7fddd40a98f0 2024-03-25T18:16:30.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.164+0000 7fddcbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:30.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.164+0000 7fddcbfff640 1 -- 172.21.15.37:0/765312486 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fddd0001c40 con 0x7fddd40a98f0 2024-03-25T18:16:30.165 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.164+0000 7fddcbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.164+0000 7fddcbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.164+0000 7fddcbfff640 10 monclient: dump: 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.164+0000 7fdddb996640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:30.166 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.164+0000 7fdddb996640 4 set_mon_vals no callback set 2024-03-25T18:16:30.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.165+0000 7fdddb996640 10 monclient: shutdown 2024-03-25T18:16:30.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.165+0000 7fdddb996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:30.167 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.165+0000 7fdddb996640 1 -- 172.21.15.37:0/765312486 >> v1:172.21.15.117:6789/0 conn(0x7fddd40a98f0 legacy=0x7fddd40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.165+0000 7fdddb996640 1 -- 172.21.15.37:0/765312486 shutdown_connections 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.165+0000 7fdddb996640 1 -- 172.21.15.37:0/765312486 >> 172.21.15.37:0/765312486 conn(0x7fddd40b7ea0 msgr2=0x7fddd40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.165+0000 7fdddb996640 1 -- 172.21.15.37:0/765312486 shutdown_connections 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.165+0000 7fdddb996640 1 -- 172.21.15.37:0/765312486 wait complete. 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.166+0000 7fdddb996640 10 monclient: build_initial_monmap 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.166+0000 7fdddb996640 10 monclient: monmap: 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:30.168 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:16:30.169 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:16:30.169 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:30.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.166+0000 7fdddb996640 1 librados: starting msgr at 2024-03-25T18:16:30.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.166+0000 7fdddb996640 1 librados: starting objecter 2024-03-25T18:16:30.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.166+0000 7fdddb996640 1 Processor -- start 2024-03-25T18:16:30.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.166+0000 7fdddb996640 1 -- start start 2024-03-25T18:16:30.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.166+0000 7fdddb996640 1 librados: setting wanted keys 2024-03-25T18:16:30.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.166+0000 7fdddb996640 1 librados: calling monclient init 2024-03-25T18:16:30.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.166+0000 7fdddb996640 10 monclient: init 2024-03-25T18:16:30.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.167+0000 7fdddb996640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:30.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.167+0000 7fdddb996640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:30.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.167+0000 7fdddb996640 10 monclient(hunting): picked mon.noname-b con 0x7fddd40a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:30.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.167+0000 7fdddb996640 10 monclient(hunting): picked mon.noname-a con 0x7fddd413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:30.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.167+0000 7fdddb996640 10 monclient(hunting): picked mon.noname-c con 0x7fddd41430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:30.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.167+0000 7fdddb996640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd4138b20 con 0x7fddd413f710 2024-03-25T18:16:30.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.167+0000 7fdddb996640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd4137a80 con 0x7fddd40a98f0 2024-03-25T18:16:30.170 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.167+0000 7fdddb996640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fddd4139880 con 0x7fddd41430c0 2024-03-25T18:16:30.171 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.167+0000 7fdddb996640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:30.171 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.167+0000 7fdddb996640 10 monclient(hunting): authenticate will time out at 7345.420180s 2024-03-25T18:16:30.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.174+0000 7fddd9732640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fddd40a98f0 0x7fddd4137370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:51088/0 (socket says 172.21.15.37:51088) 2024-03-25T18:16:30.175 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.174+0000 7fddd9732640 1 -- 172.21.15.37:0/997545045 learned_addr learned my addr 172.21.15.37:0/997545045 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:30.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.177+0000 7fddc9ffb640 10 client.?.objecter ms_handle_connect 0x7fddd40a98f0 2024-03-25T18:16:30.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.177+0000 7fddc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:30.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.180+0000 7fddc9ffb640 10 client.?.objecter ms_handle_connect 0x7fddd41430c0 2024-03-25T18:16:30.181 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.180+0000 7fddc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:30.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.185+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 854388584 0 0) 0x7fddd4137a80 con 0x7fddd40a98f0 2024-03-25T18:16:30.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.185+0000 7fddc9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:30.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.185+0000 7fddc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:30.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.185+0000 7fddc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:30.186 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.185+0000 7fddc9ffb640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:16:30.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.185+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddb8004eb0 con 0x7fddd40a98f0 2024-03-25T18:16:30.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.186+0000 7fddc9ffb640 10 client.?.objecter ms_handle_connect 0x7fddd413f710 2024-03-25T18:16:30.187 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.186+0000 7fddc9ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:30.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.188+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3958933240 0 0) 0x7fddd4139880 con 0x7fddd41430c0 2024-03-25T18:16:30.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.188+0000 7fddc9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:30.189 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.188+0000 7fddc9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:30.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.188+0000 7fddc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:30.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.188+0000 7fddc9ffb640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:16:30.190 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.188+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddd4137a80 con 0x7fddd41430c0 2024-03-25T18:16:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.196+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2304429683 0 0) 0x7fddb8004eb0 con 0x7fddd40a98f0 2024-03-25T18:16:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.196+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddd4139880 con 0x7fddd40a98f0 2024-03-25T18:16:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.196+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fddd0001bc0 con 0x7fddd40a98f0 2024-03-25T18:16:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.196+0000 7fddc9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.196+0000 7fddc9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:30.197 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.196+0000 7fddc9ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:30.197 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:30.197 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:30.197 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:30.198 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:30.198 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.198 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:30.198 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:30.198 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:30.198 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:30.198 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:30.198 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.196+0000 7fddc9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.198+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1573242896 0 0) 0x7fddd4137a80 con 0x7fddd41430c0 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.198+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fddb8004eb0 con 0x7fddd41430c0 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.198+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fddcc001b10 con 0x7fddd41430c0 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.200 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:30.201 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:30.201 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:30.201 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:30.201 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:30.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1363052607 0 0) 0x7fddd4139880 con 0x7fddd40a98f0 2024-03-25T18:16:30.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 >> v1:172.21.15.37:6790/0 conn(0x7fddd41430c0 legacy=0x7fddd41e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 >> v1:172.21.15.37:6789/0 conn(0x7fddd413f710 legacy=0x7fddd4139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:30.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 1 monclient: found mon.b 2024-03-25T18:16:30.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:30.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:30.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd4139a10 con 0x7fddd40a98f0 2024-03-25T18:16:30.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:30.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:30.202 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fddc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fdddb996640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fdddb996640 10 monclient: _renew_subs 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fdddb996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fdddb996640 1 -- 172.21.15.37:0/997545045 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd41dab00 con 0x7fddd40a98f0 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fdddb996640 10 client.4621.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fdddb996640 10 monclient: _renew_subs 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fdddb996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fdddb996640 1 -- 172.21.15.37:0/997545045 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd413b4f0 con 0x7fddd40a98f0 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.199+0000 7fdddb996640 1 librados: init done 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.200+0000 7fdddb996640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.200+0000 7fdddb996640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.200+0000 7fdddb996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:30.203 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.200+0000 7fdddb996640 1 -- 172.21.15.37:0/997545045 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd41dae00 con 0x7fddd40a98f0 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.474+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fddd0001d20 con 0x7fddd40a98f0 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.474+0000 7fddc9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.474+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fddd00010e0 con 0x7fddd40a98f0 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.474+0000 7fddc9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.474+0000 7fddc9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.474+0000 7fddc9ffb640 10 monclient: dump: 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:30.476 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:30.477 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:30.477 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:30.477 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:30.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.474+0000 7fddcb7fe640 4 set_mon_vals no callback set 2024-03-25T18:16:30.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.475+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fddd001ac40 con 0x7fddd40a98f0 2024-03-25T18:16:30.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.476+0000 7fddd8f31640 10 monclient: get_auth_request con 0x7fddb806dea0 auth_method 0 2024-03-25T18:16:30.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.476+0000 7fddc9ffb640 10 client.4621.objecter ms_handle_connect 0x7fddb806dea0 2024-03-25T18:16:30.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.480+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fddd0054940 con 0x7fddd40a98f0 2024-03-25T18:16:30.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.480+0000 7fddc9ffb640 10 client.4621.objecter ms_dispatch 0x7fddd4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:30.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.480+0000 7fddc9ffb640 3 client.4621.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:30.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.480+0000 7fddc9ffb640 3 client.4621.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:30.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.480+0000 7fddc9ffb640 20 client.4621.objecter dump_active .. 0 homeless 2024-03-25T18:16:30.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.481+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fddd0050280 con 0x7fddd40a98f0 2024-03-25T18:16:30.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.481+0000 7fddc9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:30.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.481+0000 7fddc9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:30.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.714+0000 7fdddb996640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:30.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.714+0000 7fdddb996640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:30.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.714+0000 7fdddb996640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:30.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.714+0000 7fdddb996640 1 -- 172.21.15.37:0/997545045 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fddd40b3e80 con 0x7fddd40a98f0 2024-03-25T18:16:30.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.737+0000 7fddc9ffb640 1 -- 172.21.15.37:0/997545045 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fddd0052380 con 0x7fddd40a98f0 2024-03-25T18:16:30.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.737+0000 7fddc9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:30.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.737+0000 7fddc9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:30.739 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:16:30.739 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:30.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 10 librados: watch_flush enter 2024-03-25T18:16:30.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 10 librados: watch_flush exit 2024-03-25T18:16:30.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 20 client.4621.objecter shutdown clearing up homeless session... 2024-03-25T18:16:30.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 10 client.4621.objecter successfully canceled tick 2024-03-25T18:16:30.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 1 -- 172.21.15.37:0/997545045 >> v1:172.21.15.117:6800/3703993921 conn(0x7fddb806dea0 legacy=0x7fddb8070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 10 monclient: shutdown 2024-03-25T18:16:30.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:30.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 1 -- 172.21.15.37:0/997545045 >> v1:172.21.15.117:6789/0 conn(0x7fddd40a98f0 legacy=0x7fddd4137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:30.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 1 -- 172.21.15.37:0/997545045 shutdown_connections 2024-03-25T18:16:30.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 1 -- 172.21.15.37:0/997545045 >> 172.21.15.37:0/997545045 conn(0x7fddd40b7ea0 msgr2=0x7fddd4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:30.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 1 -- 172.21.15.37:0/997545045 shutdown_connections 2024-03-25T18:16:30.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 1 -- 172.21.15.37:0/997545045 wait complete. 2024-03-25T18:16:30.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:30.740+0000 7fdddb996640 1 librados: shutdown 2024-03-25T18:16:30.753 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:30.769 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:09:06.114 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:30.825 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:09:06.170 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:30.855 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:09:06.200 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:30.869 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:09:06.182 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:30.876 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:09:06.190 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:30.880 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:09:06.225 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:30.891 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:09:06.236 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:30.916 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:09:06.229 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:30.921 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:09:06.266 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:37.558 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.673+0000 7f38d90b3640 10 monclient: get_monmap_and_config 2024-03-25T18:16:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.673+0000 7f38d90b3640 10 monclient: build_initial_monmap 2024-03-25T18:16:37.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.673+0000 7f38d90b3640 10 monclient: monmap: 2024-03-25T18:16:37.675 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:37.675 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:37.675 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:16:37.674864+0000 2024-03-25T18:16:37.675 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:16:37.674864+0000 2024-03-25T18:16:37.675 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:37.675 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:37.675 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:37.676 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:16:37.676 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:16:37.676 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:37.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.674+0000 7f38d90b3640 1 Processor -- start 2024-03-25T18:16:37.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.674+0000 7f38d90b3640 1 -- start start 2024-03-25T18:16:37.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.674+0000 7f38d90b3640 10 monclient: init 2024-03-25T18:16:37.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.675+0000 7f38d90b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:37.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.675+0000 7f38d90b3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:37.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.675+0000 7f38d90b3640 10 monclient(hunting): picked mon.noname-a con 0x7f38d40a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:37.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.675+0000 7f38d90b3640 10 monclient(hunting): picked mon.noname-c con 0x7f38d413f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:37.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.675+0000 7f38d90b3640 10 monclient(hunting): picked mon.noname-b con 0x7f38d41430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:37.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.675+0000 7f38d90b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d4142520 con 0x7f38d40a98f0 2024-03-25T18:16:37.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.675+0000 7f38d90b3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d40b28a0 con 0x7f38d413f710 2024-03-25T18:16:37.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.675+0000 7f38d90b3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d40b2a30 con 0x7f38d41430c0 2024-03-25T18:16:37.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.675+0000 7f38d90b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:37.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.675+0000 7f38d90b3640 10 monclient(hunting): authenticate will time out at 7352.927993s 2024-03-25T18:16:37.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.677+0000 7f38d259c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f38d413f710 0x7f38d41429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:49254/0 (socket says 172.21.15.37:49254) 2024-03-25T18:16:37.678 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.677+0000 7f38d259c640 1 -- 172.21.15.37:0/643744677 learned_addr learned my addr 172.21.15.37:0/643744677 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:37.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.687+0000 7f38d1d9b640 1 -- 172.21.15.37:0/643744677 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2054112342 0 0) 0x7f38d40b28a0 con 0x7f38d413f710 2024-03-25T18:16:37.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.687+0000 7f38d1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.687+0000 7f38d1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.687+0000 7f38d1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.687+0000 7f38d1d9b640 10 monclient(hunting): my global_id is 4624 2024-03-25T18:16:37.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.687+0000 7f38d1d9b640 1 -- 172.21.15.37:0/643744677 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38bc003520 con 0x7f38d413f710 2024-03-25T18:16:37.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 1 -- 172.21.15.37:0/643744677 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 282816758 0 0) 0x7f38bc003520 con 0x7f38d413f710 2024-03-25T18:16:37.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 1 -- 172.21.15.37:0/643744677 >> v1:172.21.15.37:6790/0 conn(0x7f38d41430c0 legacy=0x7f38d414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:37.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 1 -- 172.21.15.37:0/643744677 >> v1:172.21.15.37:6789/0 conn(0x7f38d40a98f0 legacy=0x7f38d40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:16:37.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 1 monclient: found mon.noname-c 2024-03-25T18:16:37.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:37.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:16:37.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 1 -- 172.21.15.37:0/643744677 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d40b44c0 con 0x7f38d413f710 2024-03-25T18:16:37.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 1 -- 172.21.15.37:0/643744677 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f38c8001690 con 0x7f38d413f710 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d90b3640 5 monclient: authenticate success, global_id 4624 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d90b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d1d9b640 10 monclient: dump: 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:37.696 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:37.697 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:37.697 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:37.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.694+0000 7f38d90b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.698+0000 7f38d1d9b640 1 -- 172.21.15.37:0/643744677 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38c8001860 con 0x7f38d413f710 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.698+0000 7f38d1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.698+0000 7f38d1d9b640 1 -- 172.21.15.37:0/643744677 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f38c8001cb0 con 0x7f38d413f710 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.698+0000 7f38d1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.698+0000 7f38d1d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.698+0000 7f38d1d9b640 10 monclient: dump: 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:37.699 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:37.700 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:37.700 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:37.700 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:37.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.698+0000 7f38d90b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:37.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.698+0000 7f38d90b3640 4 set_mon_vals no callback set 2024-03-25T18:16:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.698+0000 7f38d90b3640 10 monclient: shutdown 2024-03-25T18:16:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.698+0000 7f38d90b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.698+0000 7f38d90b3640 1 -- 172.21.15.37:0/643744677 >> v1:172.21.15.117:6789/0 conn(0x7f38d413f710 legacy=0x7f38d41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.699+0000 7f38d90b3640 1 -- 172.21.15.37:0/643744677 shutdown_connections 2024-03-25T18:16:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.699+0000 7f38d90b3640 1 -- 172.21.15.37:0/643744677 >> 172.21.15.37:0/643744677 conn(0x7f38d40b7ea0 msgr2=0x7f38d40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.699+0000 7f38d90b3640 1 -- 172.21.15.37:0/643744677 shutdown_connections 2024-03-25T18:16:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.699+0000 7f38d90b3640 1 -- 172.21.15.37:0/643744677 wait complete. 2024-03-25T18:16:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.699+0000 7f38d90b3640 10 monclient: build_initial_monmap 2024-03-25T18:16:37.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.700+0000 7f38d90b3640 10 monclient: monmap: 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.700+0000 7f38d90b3640 1 librados: starting msgr at 2024-03-25T18:16:37.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.700+0000 7f38d90b3640 1 librados: starting objecter 2024-03-25T18:16:37.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.700+0000 7f38d90b3640 1 Processor -- start 2024-03-25T18:16:37.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.700+0000 7f38d90b3640 1 -- start start 2024-03-25T18:16:37.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.701+0000 7f38d90b3640 1 librados: setting wanted keys 2024-03-25T18:16:37.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.701+0000 7f38d90b3640 1 librados: calling monclient init 2024-03-25T18:16:37.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.701+0000 7f38d90b3640 10 monclient: init 2024-03-25T18:16:37.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.702+0000 7f38d90b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:37.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.702+0000 7f38d90b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:37.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.702+0000 7f38d90b3640 10 monclient(hunting): picked mon.noname-a con 0x7f38d40a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:37.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.702+0000 7f38d90b3640 10 monclient(hunting): picked mon.noname-c con 0x7f38d413f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:37.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.702+0000 7f38d90b3640 10 monclient(hunting): picked mon.noname-b con 0x7f38d41430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:37.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.702+0000 7f38d90b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d41359a0 con 0x7f38d40a98f0 2024-03-25T18:16:37.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.702+0000 7f38d90b3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d41319b0 con 0x7f38d41430c0 2024-03-25T18:16:37.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.702+0000 7f38d90b3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38d40b7bb0 con 0x7f38d413f710 2024-03-25T18:16:37.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.702+0000 7f38d90b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:37.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.702+0000 7f38d90b3640 10 monclient(hunting): authenticate will time out at 7352.954890s 2024-03-25T18:16:37.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.712+0000 7f38d359e640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f38d41430c0 0x7f38d41312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:49276/0 (socket says 172.21.15.37:49276) 2024-03-25T18:16:37.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.712+0000 7f38d359e640 1 -- 172.21.15.37:0/591577415 learned_addr learned my addr 172.21.15.37:0/591577415 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:37.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.718+0000 7f38bb7fe640 10 client.?.objecter ms_handle_connect 0x7f38d41430c0 2024-03-25T18:16:37.719 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.718+0000 7f38bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:37.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.722+0000 7f38bb7fe640 10 client.?.objecter ms_handle_connect 0x7f38d40a98f0 2024-03-25T18:16:37.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.722+0000 7f38bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:37.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.722+0000 7f38bb7fe640 10 client.?.objecter ms_handle_connect 0x7f38d413f710 2024-03-25T18:16:37.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.722+0000 7f38bb7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:37.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.727+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3937982293 0 0) 0x7f38d40b7bb0 con 0x7f38d413f710 2024-03-25T18:16:37.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.727+0000 7f38bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.727+0000 7f38bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.727+0000 7f38bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.727+0000 7f38bb7fe640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:16:37.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.727+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38c0004eb0 con 0x7f38d413f710 2024-03-25T18:16:37.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.734+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2159017049 0 0) 0x7f38d41359a0 con 0x7f38d40a98f0 2024-03-25T18:16:37.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.734+0000 7f38bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.734+0000 7f38bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.734+0000 7f38bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.734+0000 7f38bb7fe640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:16:37.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.734+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d40b7bb0 con 0x7f38d40a98f0 2024-03-25T18:16:37.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.737+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1478999315 0 0) 0x7f38d41319b0 con 0x7f38d41430c0 2024-03-25T18:16:37.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.737+0000 7f38bb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:37.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.737+0000 7f38bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:37.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.737+0000 7f38bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:37.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.737+0000 7f38bb7fe640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:16:37.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.737+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38d41359a0 con 0x7f38d41430c0 2024-03-25T18:16:37.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.738+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2677824565 0 0) 0x7f38c0004eb0 con 0x7f38d413f710 2024-03-25T18:16:37.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.738+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38d41319b0 con 0x7f38d413f710 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.738+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f38c8001b70 con 0x7f38d413f710 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.738+0000 7f38bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.738+0000 7f38bb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.738+0000 7f38bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:37.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.738+0000 7f38bb7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.742+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4176006584 0 0) 0x7f38d41359a0 con 0x7f38d41430c0 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.742+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f38c0004eb0 con 0x7f38d41430c0 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.743+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f38cc001a10 con 0x7f38d41430c0 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.743+0000 7f38bb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.743+0000 7f38bb7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.743+0000 7f38bb7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:37.744 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:37.745 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:37.745 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:37.745 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:37.745 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:37.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1421502057 0 0) 0x7f38d41319b0 con 0x7f38d413f710 2024-03-25T18:16:37.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 >> v1:172.21.15.117:6789/0 conn(0x7f38d41430c0 legacy=0x7f38d41312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 >> v1:172.21.15.37:6789/0 conn(0x7f38d40a98f0 legacy=0x7f38d41d7600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:37.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38bb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:37.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38bb7fe640 1 monclient: found mon.c 2024-03-25T18:16:37.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:37.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38bb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:37.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d4135070 con 0x7f38d413f710 2024-03-25T18:16:37.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38bb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:37.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38bb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:37.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:37.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38d90b3640 5 monclient: authenticate success, global_id 4622 2024-03-25T18:16:37.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38d90b3640 10 monclient: _renew_subs 2024-03-25T18:16:37.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38d90b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:37.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38d90b3640 1 -- 172.21.15.37:0/591577415 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d413b400 con 0x7f38d413f710 2024-03-25T18:16:37.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38d90b3640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:37.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38d90b3640 10 monclient: _renew_subs 2024-03-25T18:16:37.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38d90b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:37.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38d90b3640 1 -- 172.21.15.37:0/591577415 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d40b75a0 con 0x7f38d413f710 2024-03-25T18:16:37.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.748+0000 7f38d90b3640 1 librados: init done 2024-03-25T18:16:37.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.749+0000 7f38d90b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:37.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.749+0000 7f38d90b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:37.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.749+0000 7f38d90b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:37.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:37.749+0000 7f38d90b3640 1 -- 172.21.15.37:0/591577415 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38d414f320 con 0x7f38d413f710 2024-03-25T18:16:38.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.080+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f38c8001010 con 0x7f38d413f710 2024-03-25T18:16:38.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.080+0000 7f38bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:38.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.080+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f38c8004bf0 con 0x7f38d413f710 2024-03-25T18:16:38.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.080+0000 7f38bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:38.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.080+0000 7f38bb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:38.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.080+0000 7f38bb7fe640 10 monclient: dump: 2024-03-25T18:16:38.081 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:38.082 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:38.082 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:38.082 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:38.082 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:38.082 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:38.082 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:38.082 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:38.082 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:38.082 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:38.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.080+0000 7f38d159a640 4 set_mon_vals no callback set 2024-03-25T18:16:38.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.081+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f38c801ac00 con 0x7f38d413f710 2024-03-25T18:16:38.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.081+0000 7f38d2d9d640 10 monclient: get_auth_request con 0x7f38c006dec0 auth_method 0 2024-03-25T18:16:38.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.081+0000 7f38bb7fe640 10 client.4622.objecter ms_handle_connect 0x7f38c006dec0 2024-03-25T18:16:38.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.088+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f38c80548e0 con 0x7f38d413f710 2024-03-25T18:16:38.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.088+0000 7f38bb7fe640 10 client.4622.objecter ms_dispatch 0x7f38d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:38.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.088+0000 7f38bb7fe640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:38.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.088+0000 7f38bb7fe640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:38.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.088+0000 7f38bb7fe640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-25T18:16:38.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.088+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f38c8050220 con 0x7f38d413f710 2024-03-25T18:16:38.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.088+0000 7f38bb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:38.089 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.088+0000 7f38bb7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:38.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.367+0000 7f38d90b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:38.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.367+0000 7f38d90b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:38.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.367+0000 7f38d90b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:38.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.367+0000 7f38d90b3640 1 -- 172.21.15.37:0/591577415 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f38d4135b50 con 0x7f38d413f710 2024-03-25T18:16:38.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.406+0000 7f38bb7fe640 1 -- 172.21.15.37:0/591577415 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f38d4135b50 con 0x7f38d413f710 2024-03-25T18:16:38.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.406+0000 7f38bb7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:38.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.406+0000 7f38bb7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:38.407 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:16:38.407 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:38.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.408+0000 7f38d90b3640 10 librados: watch_flush enter 2024-03-25T18:16:38.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.408+0000 7f38d90b3640 10 librados: watch_flush exit 2024-03-25T18:16:38.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.408+0000 7f38d90b3640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-25T18:16:38.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.408+0000 7f38d90b3640 10 client.4622.objecter successfully canceled tick 2024-03-25T18:16:38.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.408+0000 7f38d90b3640 1 -- 172.21.15.37:0/591577415 >> v1:172.21.15.117:6800/3703993921 conn(0x7f38c006dec0 legacy=0x7f38c0070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:38.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.408+0000 7f38d90b3640 10 monclient: shutdown 2024-03-25T18:16:38.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.408+0000 7f38d90b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:38.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.408+0000 7f38d90b3640 1 -- 172.21.15.37:0/591577415 >> v1:172.21.15.37:6790/0 conn(0x7f38d413f710 legacy=0x7f38d4130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:38.410 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.408+0000 7f38d90b3640 1 -- 172.21.15.37:0/591577415 shutdown_connections 2024-03-25T18:16:38.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.408+0000 7f38d90b3640 1 -- 172.21.15.37:0/591577415 >> 172.21.15.37:0/591577415 conn(0x7f38d40b7ea0 msgr2=0x7f38d4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:38.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.409+0000 7f38d90b3640 1 -- 172.21.15.37:0/591577415 shutdown_connections 2024-03-25T18:16:38.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.409+0000 7f38d90b3640 1 -- 172.21.15.37:0/591577415 wait complete. 2024-03-25T18:16:38.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:38.409+0000 7f38d90b3640 1 librados: shutdown 2024-03-25T18:16:38.423 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:45.230 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:45.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.345+0000 7f15ab02a640 10 monclient: get_monmap_and_config 2024-03-25T18:16:45.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.345+0000 7f15ab02a640 10 monclient: build_initial_monmap 2024-03-25T18:16:45.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.345+0000 7f15ab02a640 10 monclient: monmap: 2024-03-25T18:16:45.346 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:45.346 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:45.346 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:16:45.346338+0000 2024-03-25T18:16:45.346 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:16:45.346338+0000 2024-03-25T18:16:45.347 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:45.347 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:45.347 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:45.347 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:16:45.347 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:16:45.347 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:45.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 1 Processor -- start 2024-03-25T18:16:45.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 1 -- start start 2024-03-25T18:16:45.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 10 monclient: init 2024-03-25T18:16:45.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:45.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:45.348 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 10 monclient(hunting): picked mon.noname-b con 0x7f15a40a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:45.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 10 monclient(hunting): picked mon.noname-c con 0x7f15a413f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:45.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 10 monclient(hunting): picked mon.noname-a con 0x7f15a41430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:45.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a4142520 con 0x7f15a41430c0 2024-03-25T18:16:45.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40b28a0 con 0x7f15a413f710 2024-03-25T18:16:45.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40b2a30 con 0x7f15a40a98f0 2024-03-25T18:16:45.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:45.349 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.346+0000 7f15ab02a640 10 monclient(hunting): authenticate will time out at 7360.599262s 2024-03-25T18:16:45.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.361+0000 7f15a8dc6640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f15a40a98f0 0x7f15a40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:43866/0 (socket says 172.21.15.37:43866) 2024-03-25T18:16:45.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.362+0000 7f15a8dc6640 1 -- 172.21.15.37:0/3375215076 learned_addr learned my addr 172.21.15.37:0/3375215076 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:45.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.370+0000 7f15a37fe640 1 -- 172.21.15.37:0/3375215076 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3150789531 0 0) 0x7f15a4142520 con 0x7f15a41430c0 2024-03-25T18:16:45.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.370+0000 7f15a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:45.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.370+0000 7f15a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:45.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.370+0000 7f15a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:45.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.370+0000 7f15a37fe640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:16:45.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.370+0000 7f15a37fe640 1 -- 172.21.15.37:0/3375215076 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1594003520 con 0x7f15a41430c0 2024-03-25T18:16:45.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.379+0000 7f15a37fe640 1 -- 172.21.15.37:0/3375215076 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4248677897 0 0) 0x7f15a40b2a30 con 0x7f15a40a98f0 2024-03-25T18:16:45.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.379+0000 7f15a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:45.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.379+0000 7f15a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:45.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.379+0000 7f15a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:45.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.379+0000 7f15a37fe640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:16:45.381 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.379+0000 7f15a37fe640 1 -- 172.21.15.37:0/3375215076 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a4142520 con 0x7f15a40a98f0 2024-03-25T18:16:45.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 1 -- 172.21.15.37:0/3375215076 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 224920999 0 0) 0x7f1594003520 con 0x7f15a41430c0 2024-03-25T18:16:45.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 1 -- 172.21.15.37:0/3375215076 >> v1:172.21.15.37:6790/0 conn(0x7f15a40a98f0 legacy=0x7f15a40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:45.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 1 -- 172.21.15.37:0/3375215076 >> v1:172.21.15.117:6789/0 conn(0x7f15a413f710 legacy=0x7f15a41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:45.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:45.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 1 monclient: found mon.noname-a 2024-03-25T18:16:45.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:45.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 1 -- 172.21.15.37:0/3375215076 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a40b44c0 con 0x7f15a41430c0 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 1 -- 172.21.15.37:0/3375215076 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f159c001a10 con 0x7f15a41430c0 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15a37fe640 10 monclient: dump: 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:45.390 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:45.391 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:45.391 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:45.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15ab02a640 5 monclient: authenticate success, global_id 4635 2024-03-25T18:16:45.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.388+0000 7f15ab02a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:45.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15a37fe640 1 -- 172.21.15.37:0/3375215076 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f159c001b70 con 0x7f15a41430c0 2024-03-25T18:16:45.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:45.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15a37fe640 1 -- 172.21.15.37:0/3375215076 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f159c001f80 con 0x7f15a41430c0 2024-03-25T18:16:45.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15ab02a640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:45.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15ab02a640 4 set_mon_vals no callback set 2024-03-25T18:16:45.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:45.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15a37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:45.403 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15a37fe640 10 monclient: dump: 2024-03-25T18:16:45.403 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:45.404 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:45.404 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:45.404 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:45.404 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:45.404 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:45.404 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:45.404 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:45.404 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:45.404 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:45.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15ab02a640 10 monclient: shutdown 2024-03-25T18:16:45.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15ab02a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:45.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15ab02a640 1 -- 172.21.15.37:0/3375215076 >> v1:172.21.15.37:6789/0 conn(0x7f15a41430c0 legacy=0x7f15a414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:45.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15ab02a640 1 -- 172.21.15.37:0/3375215076 shutdown_connections 2024-03-25T18:16:45.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.402+0000 7f15ab02a640 1 -- 172.21.15.37:0/3375215076 >> 172.21.15.37:0/3375215076 conn(0x7f15a40b7ea0 msgr2=0x7f15a40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:45.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.403+0000 7f15ab02a640 1 -- 172.21.15.37:0/3375215076 shutdown_connections 2024-03-25T18:16:45.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.403+0000 7f15ab02a640 1 -- 172.21.15.37:0/3375215076 wait complete. 2024-03-25T18:16:45.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.403+0000 7f15ab02a640 10 monclient: build_initial_monmap 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.403+0000 7f15ab02a640 10 monclient: monmap: 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 1 librados: starting msgr at 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 1 librados: starting objecter 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 1 Processor -- start 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 1 -- start start 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 1 librados: setting wanted keys 2024-03-25T18:16:45.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 1 librados: calling monclient init 2024-03-25T18:16:45.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 10 monclient: init 2024-03-25T18:16:45.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:45.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:45.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 10 monclient(hunting): picked mon.noname-a con 0x7f15a40a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:45.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 10 monclient(hunting): picked mon.noname-c con 0x7f15a413f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:45.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 10 monclient(hunting): picked mon.noname-b con 0x7f15a41430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:45.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a4136a50 con 0x7f15a40a98f0 2024-03-25T18:16:45.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.404+0000 7f15ab02a640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40b79c0 con 0x7f15a41430c0 2024-03-25T18:16:45.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.405+0000 7f15ab02a640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f15a40b7ca0 con 0x7f15a413f710 2024-03-25T18:16:45.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.405+0000 7f15ab02a640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:45.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.405+0000 7f15ab02a640 10 monclient(hunting): authenticate will time out at 7360.657304s 2024-03-25T18:16:45.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.411+0000 7f15a95c7640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f15a41430c0 0x7f15a41e26b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:54082/0 (socket says 172.21.15.37:54082) 2024-03-25T18:16:45.412 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.411+0000 7f15a95c7640 1 -- 172.21.15.37:0/1432060257 learned_addr learned my addr 172.21.15.37:0/1432060257 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:45.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.412+0000 7f15a17fa640 10 client.?.objecter ms_handle_connect 0x7f15a41430c0 2024-03-25T18:16:45.413 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.412+0000 7f15a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:45.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.419+0000 7f15a17fa640 10 client.?.objecter ms_handle_connect 0x7f15a413f710 2024-03-25T18:16:45.420 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.419+0000 7f15a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:45.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.422+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088088544 0 0) 0x7f15a40b79c0 con 0x7f15a41430c0 2024-03-25T18:16:45.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.422+0000 7f15a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:45.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.422+0000 7f15a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:45.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.422+0000 7f15a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:45.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.422+0000 7f15a17fa640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:16:45.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.422+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1588004eb0 con 0x7f15a41430c0 2024-03-25T18:16:45.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.428+0000 7f15a17fa640 10 client.?.objecter ms_handle_connect 0x7f15a40a98f0 2024-03-25T18:16:45.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.428+0000 7f15a17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1720583041 0 0) 0x7f1588004eb0 con 0x7f15a41430c0 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f15a40b79c0 con 0x7f15a41430c0 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f159c001f80 con 0x7f15a41430c0 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): dump: 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:45.439 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019832518 0 0) 0x7f15a4136a50 con 0x7f15a40a98f0 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1588005680 con 0x7f15a40a98f0 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3779993297 0 0) 0x7f15a40b7ca0 con 0x7f15a413f710 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:45.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:16:45.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.437+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f15a4136a50 con 0x7f15a413f710 2024-03-25T18:16:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 70653241 0 0) 0x7f15a40b79c0 con 0x7f15a41430c0 2024-03-25T18:16:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 >> v1:172.21.15.37:6790/0 conn(0x7f15a413f710 legacy=0x7f15a4135350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 >> v1:172.21.15.37:6789/0 conn(0x7f15a40a98f0 legacy=0x7f15a4137680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15a17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15a17fa640 1 monclient: found mon.b 2024-03-25T18:16:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15a17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15a17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a4136ca0 con 0x7f15a41430c0 2024-03-25T18:16:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15a17fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15a17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:45.450 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15a17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15ab02a640 5 monclient: authenticate success, global_id 4633 2024-03-25T18:16:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15ab02a640 10 monclient: _renew_subs 2024-03-25T18:16:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15ab02a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15ab02a640 1 -- 172.21.15.37:0/1432060257 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15a4136f80 con 0x7f15a41430c0 2024-03-25T18:16:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15ab02a640 10 client.4633.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15ab02a640 10 monclient: _renew_subs 2024-03-25T18:16:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15ab02a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15ab02a640 1 -- 172.21.15.37:0/1432060257 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f15a41360e0 con 0x7f15a41430c0 2024-03-25T18:16:45.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.448+0000 7f15ab02a640 1 librados: init done 2024-03-25T18:16:45.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.449+0000 7f15ab02a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:45.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.449+0000 7f15ab02a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:45.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.449+0000 7f15ab02a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:45.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.449+0000 7f15ab02a640 1 -- 172.21.15.37:0/1432060257 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15a40627b0 con 0x7f15a41430c0 2024-03-25T18:16:45.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.719+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f159c003c20 con 0x7f15a41430c0 2024-03-25T18:16:45.720 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.719+0000 7f15a17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.719+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f159c004030 con 0x7f15a41430c0 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.719+0000 7f15a17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.719+0000 7f15a17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.719+0000 7f15a17fa640 10 monclient: dump: 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:45.721 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.719+0000 7f15a2ffd640 4 set_mon_vals no callback set 2024-03-25T18:16:45.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.720+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f159c004250 con 0x7f15a41430c0 2024-03-25T18:16:45.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.720+0000 7f15a8dc6640 10 monclient: get_auth_request con 0x7f158806dfa0 auth_method 0 2024-03-25T18:16:45.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.720+0000 7f15a17fa640 10 client.4633.objecter ms_handle_connect 0x7f158806dfa0 2024-03-25T18:16:45.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.724+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f159c054c10 con 0x7f15a41430c0 2024-03-25T18:16:45.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.724+0000 7f15a17fa640 10 client.4633.objecter ms_dispatch 0x7f15a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:45.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.724+0000 7f15a17fa640 3 client.4633.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:45.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.724+0000 7f15a17fa640 3 client.4633.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:45.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.724+0000 7f15a17fa640 20 client.4633.objecter dump_active .. 0 homeless 2024-03-25T18:16:45.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.724+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f159c050720 con 0x7f15a41430c0 2024-03-25T18:16:45.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.724+0000 7f15a17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:45.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.724+0000 7f15a17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:45.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.972+0000 7f15ab02a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:45.973 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.972+0000 7f15ab02a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:45.974 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.972+0000 7f15ab02a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:16:45.974 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:45.972+0000 7f15ab02a640 1 -- 172.21.15.37:0/1432060257 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f15a413e290 con 0x7f15a41430c0 2024-03-25T18:16:46.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.002+0000 7f15a17fa640 1 -- 172.21.15.37:0/1432060257 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f159c052650 con 0x7f15a41430c0 2024-03-25T18:16:46.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.002+0000 7f15a17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:46.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.002+0000 7f15a17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:46.003 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:16:46.003 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:46.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.004+0000 7f15ab02a640 10 librados: watch_flush enter 2024-03-25T18:16:46.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.004+0000 7f15ab02a640 10 librados: watch_flush exit 2024-03-25T18:16:46.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.004+0000 7f15ab02a640 20 client.4633.objecter shutdown clearing up homeless session... 2024-03-25T18:16:46.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.004+0000 7f15ab02a640 10 client.4633.objecter successfully canceled tick 2024-03-25T18:16:46.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.005+0000 7f15ab02a640 1 -- 172.21.15.37:0/1432060257 >> v1:172.21.15.117:6800/3703993921 conn(0x7f158806dfa0 legacy=0x7f1588070460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:46.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.005+0000 7f15ab02a640 10 monclient: shutdown 2024-03-25T18:16:46.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.005+0000 7f15ab02a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:46.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.005+0000 7f15ab02a640 1 -- 172.21.15.37:0/1432060257 >> v1:172.21.15.117:6789/0 conn(0x7f15a41430c0 legacy=0x7f15a41e26b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:46.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.005+0000 7f15ab02a640 1 -- 172.21.15.37:0/1432060257 shutdown_connections 2024-03-25T18:16:46.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.005+0000 7f15ab02a640 1 -- 172.21.15.37:0/1432060257 >> 172.21.15.37:0/1432060257 conn(0x7f15a40b7ea0 msgr2=0x7f15a4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:46.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.005+0000 7f15ab02a640 1 -- 172.21.15.37:0/1432060257 shutdown_connections 2024-03-25T18:16:46.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.005+0000 7f15ab02a640 1 -- 172.21.15.37:0/1432060257 wait complete. 2024-03-25T18:16:46.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:46.005+0000 7f15ab02a640 1 librados: shutdown 2024-03-25T18:16:46.019 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:52.826 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:52.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.942+0000 7f009b0c1640 10 monclient: get_monmap_and_config 2024-03-25T18:16:52.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.942+0000 7f009b0c1640 10 monclient: build_initial_monmap 2024-03-25T18:16:52.944 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.942+0000 7f009b0c1640 10 monclient: monmap: 2024-03-25T18:16:52.944 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:52.944 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:52.944 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:16:52.944085+0000 2024-03-25T18:16:52.945 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:16:52.944085+0000 2024-03-25T18:16:52.945 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:52.945 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:52.945 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:52.945 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:16:52.945 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:16:52.945 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:52.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.943+0000 7f009b0c1640 1 Processor -- start 2024-03-25T18:16:52.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.943+0000 7f009b0c1640 1 -- start start 2024-03-25T18:16:52.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.943+0000 7f009b0c1640 10 monclient: init 2024-03-25T18:16:52.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.943+0000 7f009b0c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:52.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.943+0000 7f009b0c1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:52.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.944+0000 7f009b0c1640 10 monclient(hunting): picked mon.noname-c con 0x7f00940a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:52.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.944+0000 7f009b0c1640 10 monclient(hunting): picked mon.noname-a con 0x7f009413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:52.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.944+0000 7f009b0c1640 10 monclient(hunting): picked mon.noname-b con 0x7f00941430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:52.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.944+0000 7f009b0c1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0094142520 con 0x7f009413f710 2024-03-25T18:16:52.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.944+0000 7f009b0c1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00940b28a0 con 0x7f00940a98f0 2024-03-25T18:16:52.946 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.944+0000 7f009b0c1640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00940b2a30 con 0x7f00941430c0 2024-03-25T18:16:52.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.944+0000 7f009b0c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:52.947 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.944+0000 7f009b0c1640 10 monclient(hunting): authenticate will time out at 7368.197377s 2024-03-25T18:16:52.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.947+0000 7f009965e640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f00941430c0 0x7f009414dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:47672/0 (socket says 172.21.15.37:47672) 2024-03-25T18:16:52.949 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.947+0000 7f009965e640 1 -- 172.21.15.37:0/544417815 learned_addr learned my addr 172.21.15.37:0/544417815 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:52.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.958+0000 7f00937fe640 1 -- 172.21.15.37:0/544417815 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 581876262 0 0) 0x7f00940b2a30 con 0x7f00941430c0 2024-03-25T18:16:52.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.958+0000 7f00937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.958+0000 7f00937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.958+0000 7f00937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.958+0000 7f00937fe640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:16:52.960 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.958+0000 7f00937fe640 1 -- 172.21.15.37:0/544417815 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0084003520 con 0x7f00941430c0 2024-03-25T18:16:52.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.960+0000 7f00937fe640 1 -- 172.21.15.37:0/544417815 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3185819705 0 0) 0x7f00940b28a0 con 0x7f00940a98f0 2024-03-25T18:16:52.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.960+0000 7f00937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.960+0000 7f00937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.960+0000 7f00937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.962 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.960+0000 7f00937fe640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:16:52.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.960+0000 7f00937fe640 1 -- 172.21.15.37:0/544417815 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f00940b2a30 con 0x7f00940a98f0 2024-03-25T18:16:52.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 1 -- 172.21.15.37:0/544417815 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 306168364 0 0) 0x7f0084003520 con 0x7f00941430c0 2024-03-25T18:16:52.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 1 -- 172.21.15.37:0/544417815 >> v1:172.21.15.117:6789/0 conn(0x7f00940a98f0 legacy=0x7f00940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 1 -- 172.21.15.37:0/544417815 >> v1:172.21.15.37:6789/0 conn(0x7f009413f710 legacy=0x7f00941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 1 monclient: found mon.noname-b 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 1 -- 172.21.15.37:0/544417815 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00940b44c0 con 0x7f00941430c0 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 1 -- 172.21.15.37:0/544417815 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f008c001a10 con 0x7f00941430c0 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f009b0c1640 5 monclient: authenticate success, global_id 4631 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f009b0c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f00937fe640 10 monclient: dump: 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:52.978 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:52.979 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:52.979 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.979 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:52.979 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:52.979 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:52.979 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:52.979 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:52.979 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.975+0000 7f009b0c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:52.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.981+0000 7f00937fe640 1 -- 172.21.15.37:0/544417815 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f008c001b70 con 0x7f00941430c0 2024-03-25T18:16:52.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.981+0000 7f00937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:52.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.981+0000 7f00937fe640 1 -- 172.21.15.37:0/544417815 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f008c001f80 con 0x7f00941430c0 2024-03-25T18:16:52.983 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.981+0000 7f00937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.981+0000 7f00937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.981+0000 7f00937fe640 10 monclient: dump: 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:52.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.981+0000 7f009b0c1640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:52.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.981+0000 7f009b0c1640 4 set_mon_vals no callback set 2024-03-25T18:16:52.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.982+0000 7f009b0c1640 10 monclient: shutdown 2024-03-25T18:16:52.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.982+0000 7f009b0c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:52.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.982+0000 7f009b0c1640 1 -- 172.21.15.37:0/544417815 >> v1:172.21.15.37:6790/0 conn(0x7f00941430c0 legacy=0x7f009414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:52.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.982+0000 7f009b0c1640 1 -- 172.21.15.37:0/544417815 shutdown_connections 2024-03-25T18:16:52.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.982+0000 7f009b0c1640 1 -- 172.21.15.37:0/544417815 >> 172.21.15.37:0/544417815 conn(0x7f00940b7ea0 msgr2=0x7f00940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:52.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.982+0000 7f009b0c1640 1 -- 172.21.15.37:0/544417815 shutdown_connections 2024-03-25T18:16:52.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.982+0000 7f009b0c1640 1 -- 172.21.15.37:0/544417815 wait complete. 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.983+0000 7f009b0c1640 10 monclient: build_initial_monmap 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.983+0000 7f009b0c1640 10 monclient: monmap: 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.983+0000 7f009b0c1640 1 librados: starting msgr at 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.983+0000 7f009b0c1640 1 librados: starting objecter 2024-03-25T18:16:52.987 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.984+0000 7f009b0c1640 1 Processor -- start 2024-03-25T18:16:52.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.984+0000 7f009b0c1640 1 -- start start 2024-03-25T18:16:52.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.984+0000 7f009b0c1640 1 librados: setting wanted keys 2024-03-25T18:16:52.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.984+0000 7f009b0c1640 1 librados: calling monclient init 2024-03-25T18:16:52.988 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.984+0000 7f009b0c1640 10 monclient: init 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.984+0000 7f009b0c1640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.984+0000 7f009b0c1640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.985+0000 7f009b0c1640 10 monclient(hunting): picked mon.noname-a con 0x7f00940a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.985+0000 7f009b0c1640 10 monclient(hunting): picked mon.noname-b con 0x7f009413f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.985+0000 7f009b0c1640 10 monclient(hunting): picked mon.noname-c con 0x7f00941430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.985+0000 7f009b0c1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0094137ab0 con 0x7f00940a98f0 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.985+0000 7f009b0c1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0094138bd0 con 0x7f009413f710 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.985+0000 7f009b0c1640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0094138e40 con 0x7f00941430c0 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.985+0000 7f009b0c1640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.985+0000 7f009b0c1640 10 monclient(hunting): authenticate will time out at 7368.238390s 2024-03-25T18:16:52.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.986+0000 7f009965e640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f00941430c0 0x7f00941e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:47716/0 (socket says 172.21.15.37:47716) 2024-03-25T18:16:52.990 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.986+0000 7f009965e640 1 -- 172.21.15.37:0/3626931963 learned_addr learned my addr 172.21.15.37:0/3626931963 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:16:52.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.989+0000 7f00917fa640 10 client.?.objecter ms_handle_connect 0x7f00941430c0 2024-03-25T18:16:52.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.989+0000 7f00917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.996+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 587480044 0 0) 0x7f0094138e40 con 0x7f00941430c0 2024-03-25T18:16:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.996+0000 7f00917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.996+0000 7f00917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.996+0000 7f00917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.996+0000 7f00917fa640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:16:52.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:52.996+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0078004eb0 con 0x7f00941430c0 2024-03-25T18:16:53.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.001+0000 7f00917fa640 10 client.?.objecter ms_handle_connect 0x7f009413f710 2024-03-25T18:16:53.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.001+0000 7f00917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.007+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1729613617 0 0) 0x7f0078004eb0 con 0x7f00941430c0 2024-03-25T18:16:53.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.007+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0094138e40 con 0x7f00941430c0 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.007+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f008c001ec0 con 0x7f00941430c0 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.007+0000 7f00917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.007+0000 7f00917fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.007+0000 7f00917fa640 10 monclient(hunting): dump: 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:53.010 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.007+0000 7f00917fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.007+0000 7f00917fa640 10 client.?.objecter ms_handle_connect 0x7f00940a98f0 2024-03-25T18:16:53.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.007+0000 7f00917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 180715040 0 0) 0x7f0094138e40 con 0x7f00941430c0 2024-03-25T18:16:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 >> v1:172.21.15.117:6789/0 conn(0x7f009413f710 legacy=0x7f00941df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 >> v1:172.21.15.37:6789/0 conn(0x7f00940a98f0 legacy=0x7f0094137f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f00917fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f00917fa640 1 monclient: found mon.c 2024-03-25T18:16:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f00917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f00917fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0094139120 con 0x7f00941430c0 2024-03-25T18:16:53.017 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f00917fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f00917fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f00917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f009b0c1640 5 monclient: authenticate success, global_id 4634 2024-03-25T18:16:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f009b0c1640 10 monclient: _renew_subs 2024-03-25T18:16:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f009b0c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.014+0000 7f009b0c1640 1 -- 172.21.15.37:0/3626931963 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0094139400 con 0x7f00941430c0 2024-03-25T18:16:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.015+0000 7f009b0c1640 10 client.4634.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.015+0000 7f009b0c1640 10 monclient: _renew_subs 2024-03-25T18:16:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.015+0000 7f009b0c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.015+0000 7f009b0c1640 1 -- 172.21.15.37:0/3626931963 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f00940b7880 con 0x7f00941430c0 2024-03-25T18:16:53.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.015+0000 7f009b0c1640 1 librados: init done 2024-03-25T18:16:53.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.016+0000 7f009b0c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:53.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.016+0000 7f009b0c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:53.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.016+0000 7f009b0c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:53.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.016+0000 7f009b0c1640 1 -- 172.21.15.37:0/3626931963 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0094137810 con 0x7f00941430c0 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.363+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f008c004e10 con 0x7f00941430c0 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.363+0000 7f00917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.363+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f008c002020 con 0x7f00941430c0 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.363+0000 7f00917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.363+0000 7f00917fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.363+0000 7f00917fa640 10 monclient: dump: 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:53.366 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:16:53.367 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:16:53.367 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:16:53.367 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:16:53.367 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:16:53.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.363+0000 7f0092ffd640 4 set_mon_vals no callback set 2024-03-25T18:16:53.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.364+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f008c01af90 con 0x7f00941430c0 2024-03-25T18:16:53.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.365+0000 7f0098e5d640 10 monclient: get_auth_request con 0x7f007806dc70 auth_method 0 2024-03-25T18:16:53.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.365+0000 7f00917fa640 10 client.4634.objecter ms_handle_connect 0x7f007806dc70 2024-03-25T18:16:53.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.369+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f008c054c10 con 0x7f00941430c0 2024-03-25T18:16:53.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.369+0000 7f00917fa640 10 client.4634.objecter ms_dispatch 0x7f0094000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:53.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.369+0000 7f00917fa640 3 client.4634.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:53.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.369+0000 7f00917fa640 3 client.4634.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:53.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.369+0000 7f00917fa640 20 client.4634.objecter dump_active .. 0 homeless 2024-03-25T18:16:53.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.369+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f008c050550 con 0x7f00941430c0 2024-03-25T18:16:53.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.369+0000 7f00917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:53.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.369+0000 7f00917fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:53.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.594+0000 7f009b0c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:53.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.594+0000 7f009b0c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:53.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.594+0000 7f009b0c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:16:53.596 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.594+0000 7f009b0c1640 1 -- 172.21.15.37:0/3626931963 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f00940627b0 con 0x7f00941430c0 2024-03-25T18:16:53.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.750+0000 7f00917fa640 1 -- 172.21.15.37:0/3626931963 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f008c052650 con 0x7f00941430c0 2024-03-25T18:16:53.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.750+0000 7f00917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:53.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.750+0000 7f00917fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:53.753 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:16:53.753 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:53.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 10 librados: watch_flush enter 2024-03-25T18:16:53.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 10 librados: watch_flush exit 2024-03-25T18:16:53.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 20 client.4634.objecter shutdown clearing up homeless session... 2024-03-25T18:16:53.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 10 client.4634.objecter successfully canceled tick 2024-03-25T18:16:53.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 1 -- 172.21.15.37:0/3626931963 >> v1:172.21.15.117:6800/3703993921 conn(0x7f007806dc70 legacy=0x7f0078070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:53.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 10 monclient: shutdown 2024-03-25T18:16:53.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:53.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 1 -- 172.21.15.37:0/3626931963 >> v1:172.21.15.37:6790/0 conn(0x7f00941430c0 legacy=0x7f00941e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:53.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 1 -- 172.21.15.37:0/3626931963 shutdown_connections 2024-03-25T18:16:53.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 1 -- 172.21.15.37:0/3626931963 >> 172.21.15.37:0/3626931963 conn(0x7f00940b7ea0 msgr2=0x7f0094074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:53.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 1 -- 172.21.15.37:0/3626931963 shutdown_connections 2024-03-25T18:16:53.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.752+0000 7f009b0c1640 1 -- 172.21.15.37:0/3626931963 wait complete. 2024-03-25T18:16:53.756 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:16:53.753+0000 7f009b0c1640 1 librados: shutdown 2024-03-25T18:16:53.766 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:55.850 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:09:31.195 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:55.896 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:09:31.209 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:55.897 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:09:31.210 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:55.899 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:09:31.244 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:55.927 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:09:31.241 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:55.936 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:09:31.282 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:55.970 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:09:31.316 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:55.980 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:09:31.326 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:56.004 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:09:31.349 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:00.572 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:00.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.691+0000 7fdb8e87b640 10 monclient: get_monmap_and_config 2024-03-25T18:17:00.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.691+0000 7fdb8e87b640 10 monclient: build_initial_monmap 2024-03-25T18:17:00.693 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.691+0000 7fdb8e87b640 10 monclient: monmap: 2024-03-25T18:17:00.693 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:00.693 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:00.693 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:17:00.693021+0000 2024-03-25T18:17:00.693 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:17:00.693021+0000 2024-03-25T18:17:00.693 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:00.694 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:00.694 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:00.694 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:17:00.694 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:17:00.694 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:00.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.692+0000 7fdb8e87b640 1 Processor -- start 2024-03-25T18:17:00.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.692+0000 7fdb8e87b640 1 -- start start 2024-03-25T18:17:00.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.692+0000 7fdb8e87b640 10 monclient: init 2024-03-25T18:17:00.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.693+0000 7fdb8e87b640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:00.695 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.693+0000 7fdb8e87b640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:00.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.693+0000 7fdb8e87b640 10 monclient(hunting): picked mon.noname-b con 0x7fdb880a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:00.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.693+0000 7fdb8e87b640 10 monclient(hunting): picked mon.noname-a con 0x7fdb8813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:00.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.693+0000 7fdb8e87b640 10 monclient(hunting): picked mon.noname-c con 0x7fdb881430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:00.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.693+0000 7fdb8e87b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb88142520 con 0x7fdb8813f710 2024-03-25T18:17:00.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.693+0000 7fdb8e87b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb880b28a0 con 0x7fdb881430c0 2024-03-25T18:17:00.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.693+0000 7fdb8e87b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb880b2a30 con 0x7fdb880a98f0 2024-03-25T18:17:00.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.693+0000 7fdb8e87b640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:00.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.693+0000 7fdb8e87b640 10 monclient(hunting): authenticate will time out at 7375.946508s 2024-03-25T18:17:00.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.710+0000 7fdb8ce18640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fdb881430c0 0x7fdb8814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:45856/0 (socket says 172.21.15.37:45856) 2024-03-25T18:17:00.713 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.710+0000 7fdb8ce18640 1 -- 172.21.15.37:0/3841908297 learned_addr learned my addr 172.21.15.37:0/3841908297 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:00.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.722+0000 7fdb86ffd640 1 -- 172.21.15.37:0/3841908297 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3657503717 0 0) 0x7fdb880b2a30 con 0x7fdb880a98f0 2024-03-25T18:17:00.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.722+0000 7fdb86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.722+0000 7fdb86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.722+0000 7fdb86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.722+0000 7fdb86ffd640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:17:00.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.722+0000 7fdb86ffd640 1 -- 172.21.15.37:0/3841908297 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb70003520 con 0x7fdb880a98f0 2024-03-25T18:17:00.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.724+0000 7fdb86ffd640 1 -- 172.21.15.37:0/3841908297 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3360386683 0 0) 0x7fdb880b28a0 con 0x7fdb881430c0 2024-03-25T18:17:00.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.724+0000 7fdb86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.724+0000 7fdb86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.724+0000 7fdb86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.724+0000 7fdb86ffd640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:17:00.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.724+0000 7fdb86ffd640 1 -- 172.21.15.37:0/3841908297 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb880b2a30 con 0x7fdb881430c0 2024-03-25T18:17:00.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 1 -- 172.21.15.37:0/3841908297 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3201632130 0 0) 0x7fdb70003520 con 0x7fdb880a98f0 2024-03-25T18:17:00.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 1 -- 172.21.15.37:0/3841908297 >> v1:172.21.15.117:6789/0 conn(0x7fdb881430c0 legacy=0x7fdb8814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.731 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 1 -- 172.21.15.37:0/3841908297 >> v1:172.21.15.37:6789/0 conn(0x7fdb8813f710 legacy=0x7fdb881429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 1 monclient: found mon.noname-b 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 1 -- 172.21.15.37:0/3841908297 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb880b44c0 con 0x7fdb880a98f0 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 1 -- 172.21.15.37:0/3841908297 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fdb78001690 con 0x7fdb880a98f0 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb86ffd640 10 monclient: dump: 2024-03-25T18:17:00.732 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:00.733 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:00.733 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.733 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.733 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.733 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:00.733 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:00.733 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:00.733 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:00.733 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:00.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb8e87b640 5 monclient: authenticate success, global_id 4637 2024-03-25T18:17:00.733 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.729+0000 7fdb8e87b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.745+0000 7fdb86ffd640 1 -- 172.21.15.37:0/3841908297 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb780017f0 con 0x7fdb880a98f0 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.745+0000 7fdb86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.745+0000 7fdb86ffd640 1 -- 172.21.15.37:0/3841908297 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fdb78001c00 con 0x7fdb880a98f0 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.745+0000 7fdb86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.745+0000 7fdb86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.745+0000 7fdb86ffd640 10 monclient: dump: 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:00.748 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:00.749 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:00.749 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:00.749 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:00.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.745+0000 7fdb8e87b640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:00.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.745+0000 7fdb8e87b640 4 set_mon_vals no callback set 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.746+0000 7fdb8e87b640 10 monclient: shutdown 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.746+0000 7fdb8e87b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.746+0000 7fdb8e87b640 1 -- 172.21.15.37:0/3841908297 >> v1:172.21.15.37:6790/0 conn(0x7fdb880a98f0 legacy=0x7fdb880a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.746+0000 7fdb8e87b640 1 -- 172.21.15.37:0/3841908297 shutdown_connections 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.746+0000 7fdb8e87b640 1 -- 172.21.15.37:0/3841908297 >> 172.21.15.37:0/3841908297 conn(0x7fdb880b7ea0 msgr2=0x7fdb880b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.746+0000 7fdb8e87b640 1 -- 172.21.15.37:0/3841908297 shutdown_connections 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.746+0000 7fdb8e87b640 1 -- 172.21.15.37:0/3841908297 wait complete. 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.747+0000 7fdb8e87b640 10 monclient: build_initial_monmap 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.747+0000 7fdb8e87b640 10 monclient: monmap: 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:00.751 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.747+0000 7fdb8e87b640 1 librados: starting msgr at 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.747+0000 7fdb8e87b640 1 librados: starting objecter 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.748+0000 7fdb8e87b640 1 Processor -- start 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.748+0000 7fdb8e87b640 1 -- start start 2024-03-25T18:17:00.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.748+0000 7fdb8e87b640 1 librados: setting wanted keys 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.748+0000 7fdb8e87b640 1 librados: calling monclient init 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.748+0000 7fdb8e87b640 10 monclient: init 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.748+0000 7fdb8e87b640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.748+0000 7fdb8e87b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.749+0000 7fdb8e87b640 10 monclient(hunting): picked mon.noname-a con 0x7fdb880a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.749+0000 7fdb8e87b640 10 monclient(hunting): picked mon.noname-b con 0x7fdb8813f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.749+0000 7fdb8e87b640 10 monclient(hunting): picked mon.noname-c con 0x7fdb881430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.749+0000 7fdb8e87b640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb881db7d0 con 0x7fdb880a98f0 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.749+0000 7fdb8e87b640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb880b1420 con 0x7fdb8813f710 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.749+0000 7fdb8e87b640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb880b15b0 con 0x7fdb881430c0 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.749+0000 7fdb8e87b640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:00.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.749+0000 7fdb8e87b640 10 monclient(hunting): authenticate will time out at 7376.002445s 2024-03-25T18:17:00.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.760+0000 7fdb877fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fdb8813f710 0x7fdb880b01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:45894/0 (socket says 172.21.15.37:45894) 2024-03-25T18:17:00.762 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.760+0000 7fdb877fe640 1 -- 172.21.15.37:0/2811526731 learned_addr learned my addr 172.21.15.37:0/2811526731 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:00.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.762+0000 7fdb84ff9640 10 client.?.objecter ms_handle_connect 0x7fdb8813f710 2024-03-25T18:17:00.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.762+0000 7fdb84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:00.764 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.762+0000 7fdb84ff9640 10 client.?.objecter ms_handle_connect 0x7fdb880a98f0 2024-03-25T18:17:00.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.762+0000 7fdb84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:00.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.772+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1810046290 0 0) 0x7fdb881db7d0 con 0x7fdb880a98f0 2024-03-25T18:17:00.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.772+0000 7fdb84ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.772+0000 7fdb84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.772+0000 7fdb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.772+0000 7fdb84ff9640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:17:00.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.772+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb74004eb0 con 0x7fdb880a98f0 2024-03-25T18:17:00.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.772+0000 7fdb84ff9640 10 client.?.objecter ms_handle_connect 0x7fdb881430c0 2024-03-25T18:17:00.775 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.772+0000 7fdb84ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:00.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2164658808 0 0) 0x7fdb880b1420 con 0x7fdb8813f710 2024-03-25T18:17:00.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.782 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:17:00.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb881db7d0 con 0x7fdb8813f710 2024-03-25T18:17:00.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3234122420 0 0) 0x7fdb880b15b0 con 0x7fdb881430c0 2024-03-25T18:17:00.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:00.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:00.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:00.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:17:00.783 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.780+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb880b1420 con 0x7fdb881430c0 2024-03-25T18:17:00.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.788+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1191576734 0 0) 0x7fdb881db7d0 con 0x7fdb8813f710 2024-03-25T18:17:00.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.788+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb880b15b0 con 0x7fdb8813f710 2024-03-25T18:17:00.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.788+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fdb7c001970 con 0x7fdb8813f710 2024-03-25T18:17:00.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.788+0000 7fdb84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.790 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.788+0000 7fdb84ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.788+0000 7fdb84ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:00.791 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.788+0000 7fdb84ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:00.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.793+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3499387156 0 0) 0x7fdb74004eb0 con 0x7fdb880a98f0 2024-03-25T18:17:00.795 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.793+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb881db7d0 con 0x7fdb880a98f0 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.793+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fdb78001bc0 con 0x7fdb880a98f0 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.793+0000 7fdb84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.793+0000 7fdb84ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.793+0000 7fdb84ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:00.796 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:00.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.801+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3345826893 0 0) 0x7fdb880b1420 con 0x7fdb881430c0 2024-03-25T18:17:00.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.801+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdb74004eb0 con 0x7fdb881430c0 2024-03-25T18:17:00.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.801+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fdb80001bd0 con 0x7fdb881430c0 2024-03-25T18:17:00.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.801+0000 7fdb84ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.801+0000 7fdb84ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:00.804 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.801+0000 7fdb84ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:00.804 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:00.804 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:00.804 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.804 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.804 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.804 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:00.804 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:00.804 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:00.804 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:00.804 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:00.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 220729955 0 0) 0x7fdb880b15b0 con 0x7fdb8813f710 2024-03-25T18:17:00.805 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 >> v1:172.21.15.37:6790/0 conn(0x7fdb881430c0 legacy=0x7fdb881d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 >> v1:172.21.15.37:6789/0 conn(0x7fdb880a98f0 legacy=0x7fdb881daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb84ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb84ff9640 1 monclient: found mon.b 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb84ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb84ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb880b1740 con 0x7fdb8813f710 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb84ff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb84ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb84ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb8e87b640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb8e87b640 10 monclient: _renew_subs 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb8e87b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:17:00.806 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb8e87b640 1 -- 172.21.15.37:0/2811526731 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb881db1a0 con 0x7fdb8813f710 2024-03-25T18:17:00.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb8e87b640 10 client.4645.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:00.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb8e87b640 10 monclient: _renew_subs 2024-03-25T18:17:00.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb8e87b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:17:00.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb8e87b640 1 -- 172.21.15.37:0/2811526731 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb880b08f0 con 0x7fdb8813f710 2024-03-25T18:17:00.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.803+0000 7fdb8e87b640 1 librados: init done 2024-03-25T18:17:00.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.804+0000 7fdb8e87b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:00.807 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.804+0000 7fdb8e87b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:00.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.804+0000 7fdb8e87b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:17:00.808 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.804+0000 7fdb8e87b640 1 -- 172.21.15.37:0/2811526731 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb880627b0 con 0x7fdb8813f710 2024-03-25T18:17:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.820+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdb7c003610 con 0x7fdb8813f710 2024-03-25T18:17:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.820+0000 7fdb84ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.820+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fdb7c0039a0 con 0x7fdb8813f710 2024-03-25T18:17:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.820+0000 7fdb84ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.820+0000 7fdb84ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:00.823 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.820+0000 7fdb84ff9640 10 monclient: dump: 2024-03-25T18:17:00.823 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:00.823 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:00.823 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.823 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:00.824 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:00.824 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:00.824 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:00.824 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:00.824 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:00.824 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:00.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.820+0000 7fdb867fc640 4 set_mon_vals no callback set 2024-03-25T18:17:00.824 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.821+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fdb7c003bc0 con 0x7fdb8813f710 2024-03-25T18:17:00.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.822+0000 7fdb87fff640 10 monclient: get_auth_request con 0x7fdb7406e0f0 auth_method 0 2024-03-25T18:17:00.825 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:00.822+0000 7fdb84ff9640 10 client.4645.objecter ms_handle_connect 0x7fdb7406e0f0 2024-03-25T18:17:01.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.075+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7fdb7c054730 con 0x7fdb8813f710 2024-03-25T18:17:01.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.075+0000 7fdb84ff9640 10 client.4645.objecter ms_dispatch 0x7fdb88000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:01.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.075+0000 7fdb84ff9640 3 client.4645.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:01.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.075+0000 7fdb84ff9640 3 client.4645.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:01.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.075+0000 7fdb84ff9640 20 client.4645.objecter dump_active .. 0 homeless 2024-03-25T18:17:01.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.077+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdb7c050240 con 0x7fdb8813f710 2024-03-25T18:17:01.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.077+0000 7fdb84ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:01.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.077+0000 7fdb84ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:01.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.336+0000 7fdb8e87b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:01.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.336+0000 7fdb8e87b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:01.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.336+0000 7fdb8e87b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:17:01.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.336+0000 7fdb8e87b640 1 -- 172.21.15.37:0/2811526731 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb880abae0 con 0x7fdb8813f710 2024-03-25T18:17:01.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.364+0000 7fdb84ff9640 1 -- 172.21.15.37:0/2811526731 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7fdb7c052170 con 0x7fdb8813f710 2024-03-25T18:17:01.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.364+0000 7fdb84ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:01.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.364+0000 7fdb84ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:01.367 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:17:01.367 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:01.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 10 librados: watch_flush enter 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 10 librados: watch_flush exit 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 20 client.4645.objecter shutdown clearing up homeless session... 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 10 client.4645.objecter successfully canceled tick 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 1 -- 172.21.15.37:0/2811526731 >> v1:172.21.15.117:6800/3703993921 conn(0x7fdb7406e0f0 legacy=0x7fdb740705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 10 monclient: shutdown 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 1 -- 172.21.15.37:0/2811526731 >> v1:172.21.15.117:6789/0 conn(0x7fdb8813f710 legacy=0x7fdb880b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 1 -- 172.21.15.37:0/2811526731 shutdown_connections 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 1 -- 172.21.15.37:0/2811526731 >> 172.21.15.37:0/2811526731 conn(0x7fdb880b7ea0 msgr2=0x7fdb8813c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 1 -- 172.21.15.37:0/2811526731 shutdown_connections 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 1 -- 172.21.15.37:0/2811526731 wait complete. 2024-03-25T18:17:01.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:01.367+0000 7fdb8e87b640 1 librados: shutdown 2024-03-25T18:17:01.381 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:08.188 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:08.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.301+0000 7f8d1ede8640 10 monclient: get_monmap_and_config 2024-03-25T18:17:08.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.301+0000 7f8d1ede8640 10 monclient: build_initial_monmap 2024-03-25T18:17:08.302 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.301+0000 7f8d1ede8640 10 monclient: monmap: 2024-03-25T18:17:08.302 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:08.302 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:08.302 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:17:08.302380+0000 2024-03-25T18:17:08.302 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:17:08.302380+0000 2024-03-25T18:17:08.303 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:08.303 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:08.303 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:08.303 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:17:08.303 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:17:08.303 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:08.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.302+0000 7f8d1ede8640 1 Processor -- start 2024-03-25T18:17:08.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.302+0000 7f8d1ede8640 1 -- start start 2024-03-25T18:17:08.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.302+0000 7f8d1ede8640 10 monclient: init 2024-03-25T18:17:08.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.302+0000 7f8d1ede8640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:08.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.302+0000 7f8d1ede8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:08.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.302+0000 7f8d1ede8640 10 monclient(hunting): picked mon.noname-a con 0x7f8d180a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:08.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.303+0000 7f8d1ede8640 10 monclient(hunting): picked mon.noname-b con 0x7f8d1813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:08.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.303+0000 7f8d1ede8640 10 monclient(hunting): picked mon.noname-c con 0x7f8d181430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:08.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.303+0000 7f8d1ede8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d18142520 con 0x7f8d180a98f0 2024-03-25T18:17:08.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.303+0000 7f8d1ede8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d180b28a0 con 0x7f8d181430c0 2024-03-25T18:17:08.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.303+0000 7f8d1ede8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d180b2a30 con 0x7f8d1813f710 2024-03-25T18:17:08.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.303+0000 7f8d1ede8640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:08.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.303+0000 7f8d1ede8640 10 monclient(hunting): authenticate will time out at 7383.555473s 2024-03-25T18:17:08.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.318+0000 7f8d1d385640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f8d181430c0 0x7f8d1814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:59236/0 (socket says 172.21.15.37:59236) 2024-03-25T18:17:08.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.318+0000 7f8d1d385640 1 -- 172.21.15.37:0/2576505206 learned_addr learned my addr 172.21.15.37:0/2576505206 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:08.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.331+0000 7f8d0f7fe640 1 -- 172.21.15.37:0/2576505206 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1909565611 0 0) 0x7f8d18142520 con 0x7f8d180a98f0 2024-03-25T18:17:08.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.331+0000 7f8d0f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:08.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.331+0000 7f8d0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:08.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.331+0000 7f8d0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:08.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.331+0000 7f8d0f7fe640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:17:08.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.331+0000 7f8d0f7fe640 1 -- 172.21.15.37:0/2576505206 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d00003520 con 0x7f8d180a98f0 2024-03-25T18:17:08.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.342+0000 7f8d0f7fe640 1 -- 172.21.15.37:0/2576505206 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3756434250 0 0) 0x7f8d180b28a0 con 0x7f8d181430c0 2024-03-25T18:17:08.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.342+0000 7f8d0f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:08.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.342+0000 7f8d0f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:08.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.342+0000 7f8d0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:08.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.342+0000 7f8d0f7fe640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:17:08.344 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.342+0000 7f8d0f7fe640 1 -- 172.21.15.37:0/2576505206 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d18142520 con 0x7f8d181430c0 2024-03-25T18:17:08.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 1 -- 172.21.15.37:0/2576505206 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 948541894 0 0) 0x7f8d00003520 con 0x7f8d180a98f0 2024-03-25T18:17:08.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 1 -- 172.21.15.37:0/2576505206 >> v1:172.21.15.37:6790/0 conn(0x7f8d1813f710 legacy=0x7f8d181429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 1 -- 172.21.15.37:0/2576505206 >> v1:172.21.15.117:6789/0 conn(0x7f8d181430c0 legacy=0x7f8d1814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 1 -- 172.21.15.37:0/2576505206 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d180b44c0 con 0x7f8d180a98f0 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 1 -- 172.21.15.37:0/2576505206 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8d14001690 con 0x7f8d180a98f0 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d0f7fe640 10 monclient: dump: 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.353 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.354 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:08.354 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:08.354 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:08.354 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:08.354 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:08.354 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:08.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d1ede8640 5 monclient: authenticate success, global_id 4653 2024-03-25T18:17:08.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.351+0000 7f8d1ede8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.362+0000 7f8d0f7fe640 1 -- 172.21.15.37:0/2576505206 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d140017f0 con 0x7f8d180a98f0 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.362+0000 7f8d0f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.362+0000 7f8d0f7fe640 1 -- 172.21.15.37:0/2576505206 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8d14001c00 con 0x7f8d180a98f0 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.362+0000 7f8d1ede8640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.362+0000 7f8d1ede8640 4 set_mon_vals no callback set 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.362+0000 7f8d0f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.362+0000 7f8d0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.362+0000 7f8d0f7fe640 10 monclient: dump: 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.363 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.364 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:08.364 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:08.364 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:08.364 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:08.364 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:08.364 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:08.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.363+0000 7f8d1ede8640 10 monclient: shutdown 2024-03-25T18:17:08.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.363+0000 7f8d1ede8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:08.365 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.363+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2576505206 >> v1:172.21.15.37:6789/0 conn(0x7f8d180a98f0 legacy=0x7f8d180a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.363+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2576505206 shutdown_connections 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.363+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2576505206 >> 172.21.15.37:0/2576505206 conn(0x7f8d180b7ea0 msgr2=0x7f8d180b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.363+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2576505206 shutdown_connections 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.363+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2576505206 wait complete. 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.364+0000 7f8d1ede8640 10 monclient: build_initial_monmap 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.364+0000 7f8d1ede8640 10 monclient: monmap: 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:08.366 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:17:08.367 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:08.367 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:17:08.367 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:17:08.367 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:08.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.364+0000 7f8d1ede8640 1 librados: starting msgr at 2024-03-25T18:17:08.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.364+0000 7f8d1ede8640 1 librados: starting objecter 2024-03-25T18:17:08.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.364+0000 7f8d1ede8640 1 Processor -- start 2024-03-25T18:17:08.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.364+0000 7f8d1ede8640 1 -- start start 2024-03-25T18:17:08.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.364+0000 7f8d1ede8640 1 librados: setting wanted keys 2024-03-25T18:17:08.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.364+0000 7f8d1ede8640 1 librados: calling monclient init 2024-03-25T18:17:08.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.364+0000 7f8d1ede8640 10 monclient: init 2024-03-25T18:17:08.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.365+0000 7f8d1ede8640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:08.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.365+0000 7f8d1ede8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:08.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.365+0000 7f8d1ede8640 10 monclient(hunting): picked mon.noname-b con 0x7f8d180a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:08.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.365+0000 7f8d1ede8640 10 monclient(hunting): picked mon.noname-a con 0x7f8d1813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:08.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.365+0000 7f8d1ede8640 10 monclient(hunting): picked mon.noname-c con 0x7f8d181430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:08.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.365+0000 7f8d1ede8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d180b7ce0 con 0x7f8d1813f710 2024-03-25T18:17:08.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.365+0000 7f8d1ede8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d181319b0 con 0x7f8d180a98f0 2024-03-25T18:17:08.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.365+0000 7f8d1ede8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d180b7800 con 0x7f8d181430c0 2024-03-25T18:17:08.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.365+0000 7f8d1ede8640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:08.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.365+0000 7f8d1ede8640 10 monclient(hunting): authenticate will time out at 7383.618170s 2024-03-25T18:17:08.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.383+0000 7f8d1d385640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f8d181430c0 0x7f8d181312a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:48206/0 (socket says 172.21.15.37:48206) 2024-03-25T18:17:08.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.383+0000 7f8d1d385640 1 -- 172.21.15.37:0/2254846009 learned_addr learned my addr 172.21.15.37:0/2254846009 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:08.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.387+0000 7f8d0d7fa640 10 client.?.objecter ms_handle_connect 0x7f8d1813f710 2024-03-25T18:17:08.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.387+0000 7f8d0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:08.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.388+0000 7f8d0d7fa640 10 client.?.objecter ms_handle_connect 0x7f8d181430c0 2024-03-25T18:17:08.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.388+0000 7f8d0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:08.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.394+0000 7f8d0d7fa640 10 client.?.objecter ms_handle_connect 0x7f8d180a98f0 2024-03-25T18:17:08.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.394+0000 7f8d0d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:08.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.398+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285691391 0 0) 0x7f8d180b7ce0 con 0x7f8d1813f710 2024-03-25T18:17:08.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.398+0000 7f8d0d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:08.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.398+0000 7f8d0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:08.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.398+0000 7f8d0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:08.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.398+0000 7f8d0d7fa640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:17:08.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.398+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d04004eb0 con 0x7f8d1813f710 2024-03-25T18:17:08.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.399+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3125769323 0 0) 0x7f8d180b7800 con 0x7f8d181430c0 2024-03-25T18:17:08.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.399+0000 7f8d0d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:08.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.399+0000 7f8d0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:08.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.399+0000 7f8d0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:08.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.399+0000 7f8d0d7fa640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:17:08.401 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.399+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d180b7ce0 con 0x7f8d181430c0 2024-03-25T18:17:08.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.406+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2165392784 0 0) 0x7f8d04004eb0 con 0x7f8d1813f710 2024-03-25T18:17:08.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.406+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d180b7800 con 0x7f8d1813f710 2024-03-25T18:17:08.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.406+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8d08001930 con 0x7f8d1813f710 2024-03-25T18:17:08.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.406+0000 7f8d0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:08.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.406+0000 7f8d0d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:08.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.406+0000 7f8d0d7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:08.408 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:08.408 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:08.408 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.408 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.408 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:08.408 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:08.408 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:08.408 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:08.408 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:08.408 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:08.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.406+0000 7f8d0d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:08.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.410+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518799327 0 0) 0x7f8d181319b0 con 0x7f8d180a98f0 2024-03-25T18:17:08.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.410+0000 7f8d0d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:08.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.410+0000 7f8d0d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:08.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.410+0000 7f8d0d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:08.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.410+0000 7f8d0d7fa640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:17:08.411 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.410+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d04004eb0 con 0x7f8d180a98f0 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.417+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 212966608 0 0) 0x7f8d180b7ce0 con 0x7f8d181430c0 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.417+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d181319b0 con 0x7f8d181430c0 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.417+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8d10001c10 con 0x7f8d181430c0 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.417+0000 7f8d0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.417+0000 7f8d0d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.417+0000 7f8d0d7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:08.419 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:08.420 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:08.420 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:08.420 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:08.432 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.431+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3281560191 0 0) 0x7f8d04004eb0 con 0x7f8d180a98f0 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.431+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d180b7ce0 con 0x7f8d180a98f0 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.431+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8d14001b00 con 0x7f8d180a98f0 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.431+0000 7f8d0d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.431+0000 7f8d0d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.431+0000 7f8d0d7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:08.433 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:08.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.434+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3782548551 0 0) 0x7f8d180b7800 con 0x7f8d1813f710 2024-03-25T18:17:08.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.434+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 >> v1:172.21.15.37:6790/0 conn(0x7f8d181430c0 legacy=0x7f8d181312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:08.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.434+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 >> v1:172.21.15.117:6789/0 conn(0x7f8d180a98f0 legacy=0x7f8d181dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:08.436 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.434+0000 7f8d0d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.434+0000 7f8d0d7fa640 1 monclient: found mon.a 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.434+0000 7f8d0d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.434+0000 7f8d0d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d18136f50 con 0x7f8d1813f710 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d0d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d0d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d0d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d1ede8640 5 monclient: authenticate success, global_id 4656 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d1ede8640 10 monclient: _renew_subs 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d1ede8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2254846009 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d181350f0 con 0x7f8d1813f710 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d1ede8640 10 client.4656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d1ede8640 10 monclient: _renew_subs 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d1ede8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:17:08.437 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2254846009 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d181353a0 con 0x7f8d1813f710 2024-03-25T18:17:08.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.435+0000 7f8d1ede8640 1 librados: init done 2024-03-25T18:17:08.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.436+0000 7f8d1ede8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:08.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.436+0000 7f8d1ede8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:08.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.436+0000 7f8d1ede8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:17:08.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.436+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2254846009 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d180627b0 con 0x7f8d1813f710 2024-03-25T18:17:08.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.739+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d080035f0 con 0x7f8d1813f710 2024-03-25T18:17:08.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.739+0000 7f8d0d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.739+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f8d08003980 con 0x7f8d1813f710 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.739+0000 7f8d0d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.739+0000 7f8d0d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.739+0000 7f8d0d7fa640 10 monclient: dump: 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:08.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.739+0000 7f8d0effd640 4 set_mon_vals no callback set 2024-03-25T18:17:08.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.740+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f8d08003ba0 con 0x7f8d1813f710 2024-03-25T18:17:08.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.741+0000 7f8d1cb84640 10 monclient: get_auth_request con 0x7f8d0406e180 auth_method 0 2024-03-25T18:17:08.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.741+0000 7f8d0d7fa640 10 client.4656.objecter ms_handle_connect 0x7f8d0406e180 2024-03-25T18:17:08.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.747+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f8d08053560 con 0x7f8d1813f710 2024-03-25T18:17:08.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.747+0000 7f8d0d7fa640 10 client.4656.objecter ms_dispatch 0x7f8d18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:08.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.747+0000 7f8d0d7fa640 3 client.4656.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:08.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.747+0000 7f8d0d7fa640 3 client.4656.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:08.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.747+0000 7f8d0d7fa640 20 client.4656.objecter dump_active .. 0 homeless 2024-03-25T18:17:08.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.747+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d0804f070 con 0x7f8d1813f710 2024-03-25T18:17:08.749 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.747+0000 7f8d0d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:08.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.747+0000 7f8d0d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:08.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.976+0000 7f8d1ede8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:08.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.976+0000 7f8d1ede8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:08.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.976+0000 7f8d1ede8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:17:08.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.976+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2254846009 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d1813e270 con 0x7f8d1813f710 2024-03-25T18:17:09.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.999+0000 7f8d0d7fa640 1 -- 172.21.15.37:0/2254846009 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f8d08050fa0 con 0x7f8d1813f710 2024-03-25T18:17:09.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.999+0000 7f8d0d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:09.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:08.999+0000 7f8d0d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:09.000 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:17:09.001 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:09.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 10 librados: watch_flush enter 2024-03-25T18:17:09.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 10 librados: watch_flush exit 2024-03-25T18:17:09.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 20 client.4656.objecter shutdown clearing up homeless session... 2024-03-25T18:17:09.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 10 client.4656.objecter successfully canceled tick 2024-03-25T18:17:09.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2254846009 >> v1:172.21.15.117:6800/3703993921 conn(0x7f8d0406e180 legacy=0x7f8d04070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:09.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 10 monclient: shutdown 2024-03-25T18:17:09.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:09.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2254846009 >> v1:172.21.15.37:6789/0 conn(0x7f8d1813f710 legacy=0x7f8d18130b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:09.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2254846009 shutdown_connections 2024-03-25T18:17:09.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2254846009 >> 172.21.15.37:0/2254846009 conn(0x7f8d180b7ea0 msgr2=0x7f8d18074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:09.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2254846009 shutdown_connections 2024-03-25T18:17:09.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 1 -- 172.21.15.37:0/2254846009 wait complete. 2024-03-25T18:17:09.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:09.002+0000 7f8d1ede8640 1 librados: shutdown 2024-03-25T18:17:09.016 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:15.822 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:15.900 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:09:51.245 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:15.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.926+0000 7f52165ef640 10 monclient: get_monmap_and_config 2024-03-25T18:17:15.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.926+0000 7f52165ef640 10 monclient: build_initial_monmap 2024-03-25T18:17:15.927 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.926+0000 7f52165ef640 10 monclient: monmap: 2024-03-25T18:17:15.927 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:15.928 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:15.928 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:17:15.927634+0000 2024-03-25T18:17:15.928 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:17:15.927634+0000 2024-03-25T18:17:15.928 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:15.928 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:15.928 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:15.928 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:17:15.928 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:17:15.928 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:15.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.927+0000 7f52165ef640 1 Processor -- start 2024-03-25T18:17:15.929 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.927+0000 7f52165ef640 1 -- start start 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.927+0000 7f52165ef640 10 monclient: init 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f52165ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f52165ef640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f52165ef640 10 monclient(hunting): picked mon.noname-c con 0x7f52100b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f52165ef640 10 monclient(hunting): picked mon.noname-a con 0x7f52100b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f52165ef640 10 monclient(hunting): picked mon.noname-b con 0x7f521014be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f52165ef640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100af940 con 0x7f52100b1730 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f52165ef640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100b0a70 con 0x7f52100b4d60 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f52165ef640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52100b0d50 con 0x7f521014be90 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f52165ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f52165ef640 10 monclient(hunting): authenticate will time out at 7391.180464s 2024-03-25T18:17:15.930 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f520ffff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f52100b4d60 0x7f521013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:35548/0 (socket says 172.21.15.37:35548) 2024-03-25T18:17:15.931 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.928+0000 7f520ffff640 1 -- 172.21.15.37:0/2191024544 learned_addr learned my addr 172.21.15.37:0/2191024544 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:15.953 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:09:51.298 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:15.987 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:09:51.300 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.997+0000 7f520effd640 1 -- 172.21.15.37:0/2191024544 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2167726515 0 0) 0x7f52100af940 con 0x7f52100b1730 2024-03-25T18:17:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.997+0000 7f520effd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.997+0000 7f520effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.997+0000 7f520effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:15.998 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.997+0000 7f520effd640 10 monclient(hunting): my global_id is 4659 2024-03-25T18:17:15.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:15.997+0000 7f520effd640 1 -- 172.21.15.37:0/2191024544 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51fc003520 con 0x7f52100b1730 2024-03-25T18:17:16.001 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:09:51.346 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:16.032 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:09:51.377 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.032+0000 7f520effd640 1 -- 172.21.15.37:0/2191024544 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 106796352 0 0) 0x7f52100b0d50 con 0x7f521014be90 2024-03-25T18:17:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.032+0000 7f520effd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:16.033 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.032+0000 7f520effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:16.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.032+0000 7f520effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:16.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.032+0000 7f520effd640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:17:16.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.032+0000 7f520effd640 1 -- 172.21.15.37:0/2191024544 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f52100af940 con 0x7f521014be90 2024-03-25T18:17:16.036 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:09:51.382 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:16.046 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:09:51.359 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 1 -- 172.21.15.37:0/2191024544 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3478854000 0 0) 0x7f51fc003520 con 0x7f52100b1730 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 1 -- 172.21.15.37:0/2191024544 >> v1:172.21.15.37:6790/0 conn(0x7f521014be90 legacy=0x7f521014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 1 -- 172.21.15.37:0/2191024544 >> v1:172.21.15.117:6789/0 conn(0x7f52100b4d60 legacy=0x7f521013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 1 monclient: found mon.noname-a 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 1 -- 172.21.15.37:0/2191024544 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52100b1030 con 0x7f52100b1730 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 10 monclient: _finish_auth 0 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:16.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 1 -- 172.21.15.37:0/2191024544 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f51f8001690 con 0x7f52100b1730 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f52165ef640 5 monclient: authenticate success, global_id 4659 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f52165ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f520effd640 10 monclient: dump: 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:16.051 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:16.052 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:16.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.048+0000 7f52165ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:16.060 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:09:51.406 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:16.075 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:09:51.387 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:16.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.119+0000 7f520effd640 1 -- 172.21.15.37:0/2191024544 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f51f80017f0 con 0x7f52100b1730 2024-03-25T18:17:16.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.119+0000 7f520effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:16.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.119+0000 7f520effd640 1 -- 172.21.15.37:0/2191024544 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f51f8001c00 con 0x7f52100b1730 2024-03-25T18:17:16.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.119+0000 7f52165ef640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:16.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.119+0000 7f52165ef640 4 set_mon_vals no callback set 2024-03-25T18:17:16.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.119+0000 7f520effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:16.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.119+0000 7f520effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:16.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.119+0000 7f520effd640 10 monclient: dump: 2024-03-25T18:17:16.121 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:16.121 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:16.121 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:16.122 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:16.122 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:16.122 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:16.122 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:16.122 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:16.122 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:16.122 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:16.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.120+0000 7f52165ef640 10 monclient: shutdown 2024-03-25T18:17:16.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.120+0000 7f52165ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:16.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.120+0000 7f52165ef640 1 -- 172.21.15.37:0/2191024544 >> v1:172.21.15.37:6789/0 conn(0x7f52100b1730 legacy=0x7f52100b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:16.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.120+0000 7f52165ef640 1 -- 172.21.15.37:0/2191024544 shutdown_connections 2024-03-25T18:17:16.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.120+0000 7f52165ef640 1 -- 172.21.15.37:0/2191024544 >> 172.21.15.37:0/2191024544 conn(0x7f52100617a0 msgr2=0x7f5210061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:16.124 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.120+0000 7f52165ef640 1 -- 172.21.15.37:0/2191024544 shutdown_connections 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.120+0000 7f52165ef640 1 -- 172.21.15.37:0/2191024544 wait complete. 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.121+0000 7f52165ef640 10 monclient: build_initial_monmap 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.121+0000 7f52165ef640 10 monclient: monmap: 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:17:16.125 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:17:16.126 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:16.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.121+0000 7f52165ef640 1 librados: starting msgr at 2024-03-25T18:17:16.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.121+0000 7f52165ef640 1 librados: starting objecter 2024-03-25T18:17:16.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.121+0000 7f52165ef640 1 Processor -- start 2024-03-25T18:17:16.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 1 -- start start 2024-03-25T18:17:16.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 1 librados: setting wanted keys 2024-03-25T18:17:16.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 1 librados: calling monclient init 2024-03-25T18:17:16.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 10 monclient: init 2024-03-25T18:17:16.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:16.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:16.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 10 monclient(hunting): picked mon.noname-a con 0x7f52100b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:16.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 10 monclient(hunting): picked mon.noname-b con 0x7f52100b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:16.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 10 monclient(hunting): picked mon.noname-c con 0x7f521014be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:16.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5210137fa0 con 0x7f52100b1730 2024-03-25T18:17:16.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52101361c0 con 0x7f52100b4d60 2024-03-25T18:17:16.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5210135ca0 con 0x7f521014be90 2024-03-25T18:17:16.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:16.127 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.122+0000 7f52165ef640 10 monclient(hunting): authenticate will time out at 7391.375216s 2024-03-25T18:17:16.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.145+0000 7f520ffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f52100b1730 0x7f521012fe80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34410/0 (socket says 172.21.15.37:34410) 2024-03-25T18:17:16.146 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.145+0000 7f520ffff640 1 -- 172.21.15.37:0/1459788838 learned_addr learned my addr 172.21.15.37:0/1459788838 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:16.148 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.147+0000 7f520cff9640 10 client.?.objecter ms_handle_connect 0x7f52100b1730 2024-03-25T18:17:16.148 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.147+0000 7f520cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:16.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.155+0000 7f520cff9640 10 client.?.objecter ms_handle_connect 0x7f521014be90 2024-03-25T18:17:16.156 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.155+0000 7f520cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:16.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.157+0000 7f520cff9640 10 client.?.objecter ms_handle_connect 0x7f52100b4d60 2024-03-25T18:17:16.158 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.157+0000 7f520cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:16.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.168+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4178298267 0 0) 0x7f5210137fa0 con 0x7f52100b1730 2024-03-25T18:17:16.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.168+0000 7f520cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:16.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.168+0000 7f520cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:16.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.168+0000 7f520cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:16.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.168+0000 7f520cff9640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:17:16.169 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.168+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f51f0004eb0 con 0x7f52100b1730 2024-03-25T18:17:16.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.194+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1578064952 0 0) 0x7f51f0004eb0 con 0x7f52100b1730 2024-03-25T18:17:16.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.194+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5210137fa0 con 0x7f52100b1730 2024-03-25T18:17:16.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.194+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f52000018b0 con 0x7f52100b1730 2024-03-25T18:17:16.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.194+0000 7f520cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:16.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.194+0000 7f520cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:16.195 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.194+0000 7f520cff9640 10 monclient(hunting): dump: 2024-03-25T18:17:16.196 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:16.196 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:16.196 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:16.196 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:16.196 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:16.196 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:16.196 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:16.196 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:16.196 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:16.196 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:16.196 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.194+0000 7f520cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3709657303 0 0) 0x7f5210137fa0 con 0x7f52100b1730 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 >> v1:172.21.15.37:6790/0 conn(0x7f521014be90 legacy=0x7f52101e2f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 >> v1:172.21.15.117:6789/0 conn(0x7f52100b4d60 legacy=0x7f52101dab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f520cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f520cff9640 1 monclient: found mon.a 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f520cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f520cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5210138350 con 0x7f52100b1730 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f520cff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f520cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f520cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:16.280 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f52165ef640 5 monclient: authenticate success, global_id 4662 2024-03-25T18:17:16.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f52165ef640 10 monclient: _renew_subs 2024-03-25T18:17:16.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f52165ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:17:16.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f52165ef640 1 -- 172.21.15.37:0/1459788838 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5210136410 con 0x7f52100b1730 2024-03-25T18:17:16.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f52165ef640 10 client.4662.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:16.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f52165ef640 10 monclient: _renew_subs 2024-03-25T18:17:16.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f52165ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:17:16.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f52165ef640 1 -- 172.21.15.37:0/1459788838 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5210130770 con 0x7f52100b1730 2024-03-25T18:17:16.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.278+0000 7f52165ef640 1 librados: init done 2024-03-25T18:17:16.281 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.279+0000 7f52165ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:16.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.279+0000 7f52165ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:16.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.279+0000 7f52165ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:17:16.282 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.279+0000 7f52165ef640 1 -- 172.21.15.37:0/1459788838 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5210135e30 con 0x7f52100b1730 2024-03-25T18:17:16.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.590+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f52000047e0 con 0x7f52100b1730 2024-03-25T18:17:16.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.590+0000 7f520cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:16.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.590+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f5200004c30 con 0x7f52100b1730 2024-03-25T18:17:16.591 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.590+0000 7f520cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.590+0000 7f520cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.590+0000 7f520cff9640 10 monclient: dump: 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:16.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.590+0000 7f520e7fc640 4 set_mon_vals no callback set 2024-03-25T18:17:16.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.591+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f5200004e50 con 0x7f52100b1730 2024-03-25T18:17:16.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.591+0000 7f520f7fe640 10 monclient: get_auth_request con 0x7f51f006dc70 auth_method 0 2024-03-25T18:17:16.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.591+0000 7f520cff9640 10 client.4662.objecter ms_handle_connect 0x7f51f006dc70 2024-03-25T18:17:16.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.598+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f5200054820 con 0x7f52100b1730 2024-03-25T18:17:16.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.598+0000 7f520cff9640 10 client.4662.objecter ms_dispatch 0x7f5210000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:16.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.598+0000 7f520cff9640 3 client.4662.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:16.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.598+0000 7f520cff9640 3 client.4662.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:16.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.598+0000 7f520cff9640 20 client.4662.objecter dump_active .. 0 homeless 2024-03-25T18:17:16.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.598+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5200050330 con 0x7f52100b1730 2024-03-25T18:17:16.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.598+0000 7f520cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:16.599 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.598+0000 7f520cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:16.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.851+0000 7f52165ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:16.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.851+0000 7f52165ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:16.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.851+0000 7f52165ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:17:16.852 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.851+0000 7f52165ef640 1 -- 172.21.15.37:0/1459788838 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f52100627b0 con 0x7f52100b1730 2024-03-25T18:17:16.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.881+0000 7f520cff9640 1 -- 172.21.15.37:0/1459788838 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f5200052260 con 0x7f52100b1730 2024-03-25T18:17:16.882 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.881+0000 7f520cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:16.883 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.881+0000 7f520cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:16.883 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:17:16.883 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:16.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.883+0000 7f52165ef640 10 librados: watch_flush enter 2024-03-25T18:17:16.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.883+0000 7f52165ef640 10 librados: watch_flush exit 2024-03-25T18:17:16.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.883+0000 7f52165ef640 20 client.4662.objecter shutdown clearing up homeless session... 2024-03-25T18:17:16.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.883+0000 7f52165ef640 10 client.4662.objecter successfully canceled tick 2024-03-25T18:17:16.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.883+0000 7f52165ef640 1 -- 172.21.15.37:0/1459788838 >> v1:172.21.15.117:6800/3703993921 conn(0x7f51f006dc70 legacy=0x7f51f0070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:16.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.883+0000 7f52165ef640 10 monclient: shutdown 2024-03-25T18:17:16.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.883+0000 7f52165ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:16.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.883+0000 7f52165ef640 1 -- 172.21.15.37:0/1459788838 >> v1:172.21.15.37:6789/0 conn(0x7f52100b1730 legacy=0x7f521012fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:16.885 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.883+0000 7f52165ef640 1 -- 172.21.15.37:0/1459788838 shutdown_connections 2024-03-25T18:17:16.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.883+0000 7f52165ef640 1 -- 172.21.15.37:0/1459788838 >> 172.21.15.37:0/1459788838 conn(0x7f52100617a0 msgr2=0x7f52100b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:16.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.884+0000 7f52165ef640 1 -- 172.21.15.37:0/1459788838 shutdown_connections 2024-03-25T18:17:16.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.884+0000 7f52165ef640 1 -- 172.21.15.37:0/1459788838 wait complete. 2024-03-25T18:17:16.886 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:16.884+0000 7f52165ef640 1 librados: shutdown 2024-03-25T18:17:16.898 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:23.706 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.811+0000 7febe8788640 10 monclient: get_monmap_and_config 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.811+0000 7febe8788640 10 monclient: build_initial_monmap 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.811+0000 7febe8788640 10 monclient: monmap: 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:17:23.812684+0000 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:17:23.812684+0000 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:17:23.813 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:23.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.812+0000 7febe8788640 1 Processor -- start 2024-03-25T18:17:23.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.812+0000 7febe8788640 1 -- start start 2024-03-25T18:17:23.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.812+0000 7febe8788640 10 monclient: init 2024-03-25T18:17:23.814 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.813+0000 7febe8788640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:23.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.813+0000 7febe8788640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:23.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.813+0000 7febe8788640 10 monclient(hunting): picked mon.noname-b con 0x7febe00a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:23.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.813+0000 7febe8788640 10 monclient(hunting): picked mon.noname-a con 0x7febe013f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:23.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.813+0000 7febe8788640 10 monclient(hunting): picked mon.noname-c con 0x7febe01430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:23.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.813+0000 7febe8788640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe0142520 con 0x7febe013f710 2024-03-25T18:17:23.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.813+0000 7febe8788640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe00b28a0 con 0x7febe01430c0 2024-03-25T18:17:23.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.813+0000 7febe8788640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe00b2a30 con 0x7febe00a98f0 2024-03-25T18:17:23.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.813+0000 7febe8788640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:23.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.813+0000 7febe8788640 10 monclient(hunting): authenticate will time out at 7399.066003s 2024-03-25T18:17:23.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.818+0000 7febe5d23640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7febe013f710 0x7febe01429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:38270/0 (socket says 172.21.15.37:38270) 2024-03-25T18:17:23.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.818+0000 7febe5d23640 1 -- 172.21.15.37:0/1423041004 learned_addr learned my addr 172.21.15.37:0/1423041004 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:23.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.829+0000 7febe5522640 1 -- 172.21.15.37:0/1423041004 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 350139928 0 0) 0x7febe00b2a30 con 0x7febe00a98f0 2024-03-25T18:17:23.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.829+0000 7febe5522640 10 monclient(hunting): _negotiate 2024-03-25T18:17:23.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.829+0000 7febe5522640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:23.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.829+0000 7febe5522640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:23.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.829+0000 7febe5522640 10 monclient(hunting): my global_id is 4652 2024-03-25T18:17:23.830 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.829+0000 7febe5522640 1 -- 172.21.15.37:0/1423041004 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febcc003520 con 0x7febe00a98f0 2024-03-25T18:17:23.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.833+0000 7febe5522640 1 -- 172.21.15.37:0/1423041004 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 164644585 0 0) 0x7febe0142520 con 0x7febe013f710 2024-03-25T18:17:23.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.833+0000 7febe5522640 10 monclient(hunting): _negotiate 2024-03-25T18:17:23.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.833+0000 7febe5522640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:23.834 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.833+0000 7febe5522640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:23.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.833+0000 7febe5522640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:17:23.835 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.833+0000 7febe5522640 1 -- 172.21.15.37:0/1423041004 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe00b2a30 con 0x7febe013f710 2024-03-25T18:17:23.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 1 -- 172.21.15.37:0/1423041004 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1703851063 0 0) 0x7febcc003520 con 0x7febe00a98f0 2024-03-25T18:17:23.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 1 -- 172.21.15.37:0/1423041004 >> v1:172.21.15.117:6789/0 conn(0x7febe01430c0 legacy=0x7febe014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 1 -- 172.21.15.37:0/1423041004 >> v1:172.21.15.37:6789/0 conn(0x7febe013f710 legacy=0x7febe01429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 10 monclient: _finish_hunting 0 2024-03-25T18:17:23.843 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 1 monclient: found mon.noname-b 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 1 -- 172.21.15.37:0/1423041004 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe00b44c0 con 0x7febe00a98f0 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 10 monclient: _finish_auth 0 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 10 monclient: _check_auth_tickets 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe8788640 5 monclient: authenticate success, global_id 4652 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe8788640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 1 -- 172.21.15.37:0/1423041004 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7febdc001690 con 0x7febe00a98f0 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe5522640 10 monclient: dump: 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:23.844 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:23.845 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:23.845 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:23.845 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:23.845 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:23.845 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:23.845 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:23.845 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:23.845 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:23.845 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.842+0000 7febe8788640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.849+0000 7febe5522640 1 -- 172.21.15.37:0/1423041004 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febdc0017f0 con 0x7febe00a98f0 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.849+0000 7febe5522640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.849+0000 7febe5522640 1 -- 172.21.15.37:0/1423041004 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7febdc001c00 con 0x7febe00a98f0 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.849+0000 7febe5522640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.849+0000 7febe5522640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.849+0000 7febe5522640 10 monclient: dump: 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:23.850 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:23.851 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:23.851 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:23.851 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:23.851 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:23.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.849+0000 7febe8788640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:23.851 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.849+0000 7febe8788640 4 set_mon_vals no callback set 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.849+0000 7febe8788640 10 monclient: shutdown 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.849+0000 7febe8788640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.849+0000 7febe8788640 1 -- 172.21.15.37:0/1423041004 >> v1:172.21.15.37:6790/0 conn(0x7febe00a98f0 legacy=0x7febe00a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.850+0000 7febe8788640 1 -- 172.21.15.37:0/1423041004 shutdown_connections 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.850+0000 7febe8788640 1 -- 172.21.15.37:0/1423041004 >> 172.21.15.37:0/1423041004 conn(0x7febe00b7ea0 msgr2=0x7febe00b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.850+0000 7febe8788640 1 -- 172.21.15.37:0/1423041004 shutdown_connections 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.850+0000 7febe8788640 1 -- 172.21.15.37:0/1423041004 wait complete. 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.850+0000 7febe8788640 10 monclient: build_initial_monmap 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.851+0000 7febe8788640 10 monclient: monmap: 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:23.853 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.851+0000 7febe8788640 1 librados: starting msgr at 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.851+0000 7febe8788640 1 librados: starting objecter 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.851+0000 7febe8788640 1 Processor -- start 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.851+0000 7febe8788640 1 -- start start 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.851+0000 7febe8788640 1 librados: setting wanted keys 2024-03-25T18:17:23.854 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.851+0000 7febe8788640 1 librados: calling monclient init 2024-03-25T18:17:23.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.851+0000 7febe8788640 10 monclient: init 2024-03-25T18:17:23.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.851+0000 7febe8788640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:23.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.851+0000 7febe8788640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:23.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.851+0000 7febe8788640 10 monclient(hunting): picked mon.noname-c con 0x7febe00a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:23.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.852+0000 7febe8788640 10 monclient(hunting): picked mon.noname-b con 0x7febe013f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:23.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.852+0000 7febe8788640 10 monclient(hunting): picked mon.noname-a con 0x7febe01430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:23.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.852+0000 7febe8788640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe0135610 con 0x7febe01430c0 2024-03-25T18:17:23.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.852+0000 7febe8788640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe0137230 con 0x7febe013f710 2024-03-25T18:17:23.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.852+0000 7febe8788640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe0135cf0 con 0x7febe00a98f0 2024-03-25T18:17:23.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.852+0000 7febe8788640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:23.855 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.852+0000 7febe8788640 10 monclient(hunting): authenticate will time out at 7399.104355s 2024-03-25T18:17:23.870 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.869+0000 7febe5d23640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7febe013f710 0x7febe01dab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:33166/0 (socket says 172.21.15.37:33166) 2024-03-25T18:17:23.870 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.869+0000 7febe5d23640 1 -- 172.21.15.37:0/3748838713 learned_addr learned my addr 172.21.15.37:0/3748838713 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:23.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.870+0000 7febd6ffd640 10 client.?.objecter ms_handle_connect 0x7febe013f710 2024-03-25T18:17:23.871 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.870+0000 7febd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:23.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.877+0000 7febd6ffd640 10 client.?.objecter ms_handle_connect 0x7febe01430c0 2024-03-25T18:17:23.878 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.877+0000 7febd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:23.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.879+0000 7febd6ffd640 10 client.?.objecter ms_handle_connect 0x7febe00a98f0 2024-03-25T18:17:23.880 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.879+0000 7febd6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:23.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.882+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 146517173 0 0) 0x7febe0137230 con 0x7febe013f710 2024-03-25T18:17:23.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.882+0000 7febd6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:23.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.882+0000 7febd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:23.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.882+0000 7febd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:23.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.882+0000 7febd6ffd640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:17:23.884 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.882+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febc4004eb0 con 0x7febe013f710 2024-03-25T18:17:23.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.895+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390730709 0 0) 0x7febe0135610 con 0x7febe01430c0 2024-03-25T18:17:23.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.895+0000 7febd6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:23.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.895+0000 7febd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:23.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.895+0000 7febd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:23.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.895+0000 7febd6ffd640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:17:23.896 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.895+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe0137230 con 0x7febe01430c0 2024-03-25T18:17:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.905+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 290811254 0 0) 0x7febe0135cf0 con 0x7febe00a98f0 2024-03-25T18:17:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.905+0000 7febd6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.905+0000 7febd6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.905+0000 7febd6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.905+0000 7febd6ffd640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:17:23.906 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.905+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febe0135610 con 0x7febe00a98f0 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.907+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2860617955 0 0) 0x7febc4004eb0 con 0x7febe013f710 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.907+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febe0135cf0 con 0x7febe013f710 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.907+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7febd00019f0 con 0x7febe013f710 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.907+0000 7febd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.907+0000 7febd6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.907+0000 7febd6ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:23.909 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:23.910 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:23.910 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.907+0000 7febd6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.938+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 78323927 0 0) 0x7febe0137230 con 0x7febe01430c0 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.938+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febc4004eb0 con 0x7febe01430c0 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.938+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7febd8001b10 con 0x7febe01430c0 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.938+0000 7febd6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.938+0000 7febd6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.938+0000 7febd6ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:23.940 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:23.941 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:23.941 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:23.941 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:23.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2123366200 0 0) 0x7febe0135cf0 con 0x7febe013f710 2024-03-25T18:17:23.941 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 >> v1:172.21.15.37:6790/0 conn(0x7febe00a98f0 legacy=0x7febe0130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 >> v1:172.21.15.37:6789/0 conn(0x7febe01430c0 legacy=0x7febe01e2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febd6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febd6ffd640 1 monclient: found mon.b 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febd6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febd6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe0135e80 con 0x7febe013f710 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febd6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febd6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febd6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febe8788640 5 monclient: authenticate success, global_id 4657 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febe8788640 10 monclient: _renew_subs 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febe8788640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febe8788640 1 -- 172.21.15.37:0/3748838713 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe0131660 con 0x7febe013f710 2024-03-25T18:17:23.942 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febe8788640 10 client.4657.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:23.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febe8788640 10 monclient: _renew_subs 2024-03-25T18:17:23.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febe8788640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:17:23.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febe8788640 1 -- 172.21.15.37:0/3748838713 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7febe00b7960 con 0x7febe013f710 2024-03-25T18:17:23.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.940+0000 7febe8788640 1 librados: init done 2024-03-25T18:17:23.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.941+0000 7febe8788640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:23.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.941+0000 7febe8788640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:23.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.941+0000 7febe8788640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:17:23.943 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:23.941+0000 7febe8788640 1 -- 172.21.15.37:0/3748838713 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe00627b0 con 0x7febe013f710 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.213+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7febd00047e0 con 0x7febe013f710 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.213+0000 7febd6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.213+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7febd0004af0 con 0x7febe013f710 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.213+0000 7febd6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.213+0000 7febd6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.213+0000 7febd6ffd640 10 monclient: dump: 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:24.215 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:24.216 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:24.216 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:24.216 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:24.216 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:24.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.213+0000 7febe4d21640 4 set_mon_vals no callback set 2024-03-25T18:17:24.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.214+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7febd001ab40 con 0x7febe013f710 2024-03-25T18:17:24.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.215+0000 7febe6524640 10 monclient: get_auth_request con 0x7febc406df70 auth_method 0 2024-03-25T18:17:24.216 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.215+0000 7febd6ffd640 10 client.4657.objecter ms_handle_connect 0x7febc406df70 2024-03-25T18:17:24.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.220+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7febd0054880 con 0x7febe013f710 2024-03-25T18:17:24.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.220+0000 7febd6ffd640 10 client.4657.objecter ms_dispatch 0x7febe0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:24.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.220+0000 7febd6ffd640 3 client.4657.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:24.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.220+0000 7febd6ffd640 3 client.4657.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:24.221 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.220+0000 7febd6ffd640 20 client.4657.objecter dump_active .. 0 homeless 2024-03-25T18:17:24.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.220+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7febd00501c0 con 0x7febe013f710 2024-03-25T18:17:24.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.220+0000 7febd6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:24.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.220+0000 7febd6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:24.476 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.475+0000 7febe8788640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:24.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.475+0000 7febe8788640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:24.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.475+0000 7febe8788640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:17:24.477 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.475+0000 7febe8788640 1 -- 172.21.15.37:0/3748838713 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febe0136010 con 0x7febe013f710 2024-03-25T18:17:24.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.513+0000 7febd6ffd640 1 -- 172.21.15.37:0/3748838713 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7febe0136010 con 0x7febe013f710 2024-03-25T18:17:24.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.513+0000 7febd6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:24.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.513+0000 7febd6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:24.514 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:17:24.514 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:24.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.515+0000 7febe8788640 10 librados: watch_flush enter 2024-03-25T18:17:24.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.515+0000 7febe8788640 10 librados: watch_flush exit 2024-03-25T18:17:24.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.515+0000 7febe8788640 20 client.4657.objecter shutdown clearing up homeless session... 2024-03-25T18:17:24.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.515+0000 7febe8788640 10 client.4657.objecter successfully canceled tick 2024-03-25T18:17:24.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.515+0000 7febe8788640 1 -- 172.21.15.37:0/3748838713 >> v1:172.21.15.117:6800/3703993921 conn(0x7febc406df70 legacy=0x7febc4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:24.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.515+0000 7febe8788640 10 monclient: shutdown 2024-03-25T18:17:24.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.515+0000 7febe8788640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:24.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.515+0000 7febe8788640 1 -- 172.21.15.37:0/3748838713 >> v1:172.21.15.117:6789/0 conn(0x7febe013f710 legacy=0x7febe01dab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:24.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.515+0000 7febe8788640 1 -- 172.21.15.37:0/3748838713 shutdown_connections 2024-03-25T18:17:24.517 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.515+0000 7febe8788640 1 -- 172.21.15.37:0/3748838713 >> 172.21.15.37:0/3748838713 conn(0x7febe00b7ea0 msgr2=0x7febe0074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:24.518 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.516+0000 7febe8788640 1 -- 172.21.15.37:0/3748838713 shutdown_connections 2024-03-25T18:17:24.518 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.516+0000 7febe8788640 1 -- 172.21.15.37:0/3748838713 wait complete. 2024-03-25T18:17:24.518 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:24.516+0000 7febe8788640 1 librados: shutdown 2024-03-25T18:17:24.529 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:31.336 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:31.449 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.448+0000 7f0a06f46640 10 monclient: get_monmap_and_config 2024-03-25T18:17:31.449 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.448+0000 7f0a06f46640 10 monclient: build_initial_monmap 2024-03-25T18:17:31.449 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.448+0000 7f0a06f46640 10 monclient: monmap: 2024-03-25T18:17:31.449 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:31.449 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:31.449 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:17:31.449137+0000 2024-03-25T18:17:31.449 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:17:31.449137+0000 2024-03-25T18:17:31.450 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:31.450 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:31.450 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:31.450 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:17:31.450 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:17:31.450 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:31.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.449+0000 7f0a06f46640 1 Processor -- start 2024-03-25T18:17:31.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.449+0000 7f0a06f46640 1 -- start start 2024-03-25T18:17:31.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.449+0000 7f0a06f46640 10 monclient: init 2024-03-25T18:17:31.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.449+0000 7f0a06f46640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:31.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.449+0000 7f0a06f46640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:31.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.449+0000 7f0a06f46640 10 monclient(hunting): picked mon.noname-c con 0x7f0a000b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:31.451 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.450+0000 7f0a06f46640 10 monclient(hunting): picked mon.noname-a con 0x7f0a000b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:31.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.450+0000 7f0a06f46640 10 monclient(hunting): picked mon.noname-b con 0x7f0a0014be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:31.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.450+0000 7f0a06f46640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000af940 con 0x7f0a000b1730 2024-03-25T18:17:31.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.450+0000 7f0a06f46640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000b0a70 con 0x7f0a000b4d60 2024-03-25T18:17:31.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.450+0000 7f0a06f46640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a000b0d50 con 0x7f0a0014be90 2024-03-25T18:17:31.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.450+0000 7f0a06f46640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:31.452 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.450+0000 7f0a06f46640 10 monclient(hunting): authenticate will time out at 7406.702502s 2024-03-25T18:17:31.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.462+0000 7f0a054e3640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f0a0014be90 0x7f0a0014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:60404/0 (socket says 172.21.15.37:60404) 2024-03-25T18:17:31.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.462+0000 7f0a054e3640 1 -- 172.21.15.37:0/3640796675 learned_addr learned my addr 172.21.15.37:0/3640796675 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:31.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.482+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310380096 0 0) 0x7f0a000af940 con 0x7f0a000b1730 2024-03-25T18:17:31.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.482+0000 7f09ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:31.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.482+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:31.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.482+0000 7f09ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:31.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.482+0000 7f09ff7fe640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:17:31.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.482+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09f0003520 con 0x7f0a000b1730 2024-03-25T18:17:31.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2418948096 0 0) 0x7f0a000b0a70 con 0x7f0a000b4d60 2024-03-25T18:17:31.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:31.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:31.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:31.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:17:31.486 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a000af940 con 0x7f0a000b4d60 2024-03-25T18:17:31.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 98091492 0 0) 0x7f0a000b0d50 con 0x7f0a0014be90 2024-03-25T18:17:31.487 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:31.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:31.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:31.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:17:31.488 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.485+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a000b0a70 con 0x7f0a0014be90 2024-03-25T18:17:31.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3649595237 0 0) 0x7f09f0003520 con 0x7f0a000b1730 2024-03-25T18:17:31.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 >> v1:172.21.15.37:6790/0 conn(0x7f0a0014be90 legacy=0x7f0a0014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:31.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 >> v1:172.21.15.117:6789/0 conn(0x7f0a000b4d60 legacy=0x7f0a0013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:31.495 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 1 monclient: found mon.noname-a 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a000b1030 con 0x7f0a000b1730 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f09ec001690 con 0x7f0a000b1730 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.493+0000 7f09ff7fe640 10 monclient: dump: 2024-03-25T18:17:31.496 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:31.497 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:31.497 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:31.497 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:31.497 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:31.497 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:31.497 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:31.497 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:31.497 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:31.497 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:31.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.494+0000 7f0a06f46640 5 monclient: authenticate success, global_id 4671 2024-03-25T18:17:31.497 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.494+0000 7f0a06f46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:31.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.508+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09ec0017f0 con 0x7f0a000b1730 2024-03-25T18:17:31.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.508+0000 7f09ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:31.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.508+0000 7f09ff7fe640 1 -- 172.21.15.37:0/3640796675 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f09ec001c00 con 0x7f0a000b1730 2024-03-25T18:17:31.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.508+0000 7f0a06f46640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:31.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.508+0000 7f0a06f46640 4 set_mon_vals no callback set 2024-03-25T18:17:31.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.508+0000 7f09ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:31.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.508+0000 7f09ff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:31.509 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.508+0000 7f09ff7fe640 10 monclient: dump: 2024-03-25T18:17:31.509 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:31.510 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:31.510 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:31.510 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:31.510 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:31.510 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:31.510 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:31.510 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:31.510 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:31.510 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:31.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.508+0000 7f0a06f46640 10 monclient: shutdown 2024-03-25T18:17:31.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.508+0000 7f0a06f46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:31.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.508+0000 7f0a06f46640 1 -- 172.21.15.37:0/3640796675 >> v1:172.21.15.37:6789/0 conn(0x7f0a000b1730 legacy=0x7f0a000b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:31.512 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.509+0000 7f0a06f46640 1 -- 172.21.15.37:0/3640796675 shutdown_connections 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.509+0000 7f0a06f46640 1 -- 172.21.15.37:0/3640796675 >> 172.21.15.37:0/3640796675 conn(0x7f0a000617a0 msgr2=0x7f0a00061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.509+0000 7f0a06f46640 1 -- 172.21.15.37:0/3640796675 shutdown_connections 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.509+0000 7f0a06f46640 1 -- 172.21.15.37:0/3640796675 wait complete. 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.509+0000 7f0a06f46640 10 monclient: build_initial_monmap 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.509+0000 7f0a06f46640 10 monclient: monmap: 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:17:31.513 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 1 librados: starting msgr at 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 1 librados: starting objecter 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 1 Processor -- start 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 1 -- start start 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 1 librados: setting wanted keys 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 1 librados: calling monclient init 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 10 monclient: init 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 10 monclient(hunting): picked mon.noname-c con 0x7f0a000b1730 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 10 monclient(hunting): picked mon.noname-a con 0x7f0a000b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:31.514 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 10 monclient(hunting): picked mon.noname-b con 0x7f0a0014be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:31.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a0012d790 con 0x7f0a000b4d60 2024-03-25T18:17:31.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.510+0000 7f0a06f46640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a00136770 con 0x7f0a0014be90 2024-03-25T18:17:31.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.511+0000 7f0a06f46640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a00136a50 con 0x7f0a000b1730 2024-03-25T18:17:31.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.511+0000 7f0a06f46640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:31.515 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.511+0000 7f0a06f46640 10 monclient(hunting): authenticate will time out at 7406.763305s 2024-03-25T18:17:31.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.517+0000 7f0a04ce2640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f0a000b1730 0x7f0a00137530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:60442/0 (socket says 172.21.15.37:60442) 2024-03-25T18:17:31.519 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.517+0000 7f0a04ce2640 1 -- 172.21.15.37:0/2847350388 learned_addr learned my addr 172.21.15.37:0/2847350388 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:31.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.523+0000 7f09fd7fa640 10 client.?.objecter ms_handle_connect 0x7f0a000b1730 2024-03-25T18:17:31.524 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.523+0000 7f09fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:31.545 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.544+0000 7f09fd7fa640 10 client.?.objecter ms_handle_connect 0x7f0a0014be90 2024-03-25T18:17:31.546 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.544+0000 7f09fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:31.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.548+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4249415217 0 0) 0x7f0a00136a50 con 0x7f0a000b1730 2024-03-25T18:17:31.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.548+0000 7f09fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:31.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.548+0000 7f09fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:31.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.548+0000 7f09fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:31.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.548+0000 7f09fd7fa640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:17:31.549 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.548+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e4004eb0 con 0x7f0a000b1730 2024-03-25T18:17:31.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.553+0000 7f09fd7fa640 10 client.?.objecter ms_handle_connect 0x7f0a000b4d60 2024-03-25T18:17:31.554 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.553+0000 7f09fd7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:31.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.555+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 162305529 0 0) 0x7f0a00136770 con 0x7f0a0014be90 2024-03-25T18:17:31.556 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.555+0000 7f09fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:31.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.555+0000 7f09fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:31.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.555+0000 7f09fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:31.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.555+0000 7f09fd7fa640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:17:31.557 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.555+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a00136a50 con 0x7f0a0014be90 2024-03-25T18:17:31.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.572+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 606077663 0 0) 0x7f09e4004eb0 con 0x7f0a000b1730 2024-03-25T18:17:31.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.572+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a00136770 con 0x7f0a000b1730 2024-03-25T18:17:31.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.572+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f09f4001a30 con 0x7f0a000b1730 2024-03-25T18:17:31.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.572+0000 7f09fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:31.573 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.572+0000 7f09fd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:31.574 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.572+0000 7f09fd7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:31.574 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:31.574 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:31.574 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:31.574 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:31.574 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:31.574 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:31.574 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:31.574 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:31.574 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:31.574 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:31.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.572+0000 7f09fd7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:31.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.573+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21301278 0 0) 0x7f0a0012d790 con 0x7f0a000b4d60 2024-03-25T18:17:31.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.573+0000 7f09fd7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:17:31.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.573+0000 7f09fd7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:31.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.573+0000 7f09fd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:31.575 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.573+0000 7f09fd7fa640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:17:31.576 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.573+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f09e4004eb0 con 0x7f0a000b4d60 2024-03-25T18:17:31.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.577+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3711924608 0 0) 0x7f0a00136a50 con 0x7f0a0014be90 2024-03-25T18:17:31.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.577+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a0012d790 con 0x7f0a0014be90 2024-03-25T18:17:31.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.578+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f09f8001b50 con 0x7f0a0014be90 2024-03-25T18:17:31.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.578+0000 7f09fd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:31.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.578+0000 7f09fd7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:31.579 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.578+0000 7f09fd7fa640 10 monclient(hunting): dump: 2024-03-25T18:17:31.579 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:31.580 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:31.580 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:31.580 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:31.580 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:31.580 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:31.580 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:31.580 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:31.580 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:31.580 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 579560830 0 0) 0x7f0a00136770 con 0x7f0a000b1730 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 >> v1:172.21.15.117:6789/0 conn(0x7f0a0014be90 legacy=0x7f0a0012d080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 >> v1:172.21.15.37:6789/0 conn(0x7f0a000b4d60 legacy=0x7f0a0012c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f09fd7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f09fd7fa640 1 monclient: found mon.c 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f09fd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f09fd7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a00136d30 con 0x7f0a000b1730 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f09fd7fa640 10 monclient: _finish_auth 0 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f09fd7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f09fd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:31.592 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f0a06f46640 5 monclient: authenticate success, global_id 4661 2024-03-25T18:17:31.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f0a06f46640 10 monclient: _renew_subs 2024-03-25T18:17:31.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f0a06f46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:31.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f0a06f46640 1 -- 172.21.15.37:0/2847350388 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a0013e370 con 0x7f0a000b1730 2024-03-25T18:17:31.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f0a06f46640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:31.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f0a06f46640 10 monclient: _renew_subs 2024-03-25T18:17:31.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f0a06f46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:31.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f0a06f46640 1 -- 172.21.15.37:0/2847350388 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a0013dfa0 con 0x7f0a000b1730 2024-03-25T18:17:31.593 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.590+0000 7f0a06f46640 1 librados: init done 2024-03-25T18:17:31.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.591+0000 7f0a06f46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:31.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.591+0000 7f0a06f46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:31.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.591+0000 7f0a06f46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:31.594 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.591+0000 7f0a06f46640 1 -- 172.21.15.37:0/2847350388 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a001401f0 con 0x7f0a000b1730 2024-03-25T18:17:31.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.603+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f09f4001c70 con 0x7f0a000b1730 2024-03-25T18:17:31.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.603+0000 7f09fd7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:31.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.603+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f09f4003a60 con 0x7f0a000b1730 2024-03-25T18:17:31.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.603+0000 7f09fd7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:31.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.603+0000 7f09fd7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:31.604 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.603+0000 7f09fd7fa640 10 monclient: dump: 2024-03-25T18:17:31.604 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:31.604 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:31.605 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:31.605 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:31.605 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:31.605 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:31.605 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:31.605 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:31.605 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:31.605 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:31.605 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.603+0000 7f09feffd640 4 set_mon_vals no callback set 2024-03-25T18:17:31.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.911+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f09f4003bc0 con 0x7f0a000b1730 2024-03-25T18:17:31.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.912+0000 7f09fffff640 10 monclient: get_auth_request con 0x7f09e406dfd0 auth_method 0 2024-03-25T18:17:31.913 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.912+0000 7f09fd7fa640 10 client.4661.objecter ms_handle_connect 0x7f09e406dfd0 2024-03-25T18:17:31.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.920+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f09f4054850 con 0x7f0a000b1730 2024-03-25T18:17:31.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.920+0000 7f09fd7fa640 10 client.4661.objecter ms_dispatch 0x7f0a00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:31.921 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.920+0000 7f09fd7fa640 3 client.4661.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:31.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.920+0000 7f09fd7fa640 3 client.4661.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:31.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.920+0000 7f09fd7fa640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-25T18:17:31.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.920+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f09f4050360 con 0x7f0a000b1730 2024-03-25T18:17:31.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.920+0000 7f09fd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:31.922 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:31.920+0000 7f09fd7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:32.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.176+0000 7f0a06f46640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:32.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.176+0000 7f0a06f46640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:32.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.176+0000 7f0a06f46640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:32.178 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.176+0000 7f0a06f46640 1 -- 172.21.15.37:0/2847350388 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a000afbd0 con 0x7f0a000b1730 2024-03-25T18:17:32.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.299+0000 7f09fd7fa640 1 -- 172.21.15.37:0/2847350388 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f09f4052290 con 0x7f0a000b1730 2024-03-25T18:17:32.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.300+0000 7f09fd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:32.301 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.300+0000 7f09fd7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:32.301 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:17:32.301 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:32.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 10 librados: watch_flush enter 2024-03-25T18:17:32.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 10 librados: watch_flush exit 2024-03-25T18:17:32.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-25T18:17:32.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 10 client.4661.objecter successfully canceled tick 2024-03-25T18:17:32.304 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 1 -- 172.21.15.37:0/2847350388 >> v1:172.21.15.117:6800/3703993921 conn(0x7f09e406dfd0 legacy=0x7f09e4070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:32.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 10 monclient: shutdown 2024-03-25T18:17:32.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:32.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 1 -- 172.21.15.37:0/2847350388 >> v1:172.21.15.37:6790/0 conn(0x7f0a000b1730 legacy=0x7f0a00137530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:32.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 1 -- 172.21.15.37:0/2847350388 shutdown_connections 2024-03-25T18:17:32.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 1 -- 172.21.15.37:0/2847350388 >> 172.21.15.37:0/2847350388 conn(0x7f0a000617a0 msgr2=0x7f0a0013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:32.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 1 -- 172.21.15.37:0/2847350388 shutdown_connections 2024-03-25T18:17:32.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.302+0000 7f0a06f46640 1 -- 172.21.15.37:0/2847350388 wait complete. 2024-03-25T18:17:32.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:32.303+0000 7f0a06f46640 1 librados: shutdown 2024-03-25T18:17:32.316 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:36.065 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:10:11.410 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:36.072 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:10:11.417 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:36.113 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:10:11.426 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:36.114 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:10:11.459 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:36.122 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:10:11.435 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:36.132 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:10:11.477 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:36.137 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:10:11.482 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:36.159 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:10:11.472 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:36.166 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:10:11.511 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:39.121 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:39.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.221+0000 7f6a4a4b3640 10 monclient: get_monmap_and_config 2024-03-25T18:17:39.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.221+0000 7f6a4a4b3640 10 monclient: build_initial_monmap 2024-03-25T18:17:39.222 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.221+0000 7f6a4a4b3640 10 monclient: monmap: 2024-03-25T18:17:39.222 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:39.222 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:39.222 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:17:39.222338+0000 2024-03-25T18:17:39.223 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:17:39.222338+0000 2024-03-25T18:17:39.223 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:39.223 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:39.223 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:39.223 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:17:39.223 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:17:39.223 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.222+0000 7f6a4a4b3640 1 Processor -- start 2024-03-25T18:17:39.224 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.222+0000 7f6a4a4b3640 1 -- start start 2024-03-25T18:17:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.222+0000 7f6a4a4b3640 10 monclient: init 2024-03-25T18:17:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.223+0000 7f6a4a4b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.223+0000 7f6a4a4b3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.223+0000 7f6a4a4b3640 10 monclient(hunting): picked mon.noname-b con 0x7f6a440b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.223+0000 7f6a4a4b3640 10 monclient(hunting): picked mon.noname-c con 0x7f6a440b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.223+0000 7f6a4a4b3640 10 monclient(hunting): picked mon.noname-a con 0x7f6a4414be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.223+0000 7f6a4a4b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a440af940 con 0x7f6a4414be90 2024-03-25T18:17:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.223+0000 7f6a4a4b3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a440b0a70 con 0x7f6a440b1730 2024-03-25T18:17:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.223+0000 7f6a4a4b3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a440b0d50 con 0x7f6a440b4d60 2024-03-25T18:17:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.223+0000 7f6a4a4b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:39.225 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.223+0000 7f6a4a4b3640 10 monclient(hunting): authenticate will time out at 7414.475542s 2024-03-25T18:17:39.234 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.233+0000 7f6a48a50640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f6a4414be90 0x7f6a4414e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:34644/0 (socket says 172.21.15.37:34644) 2024-03-25T18:17:39.235 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.234+0000 7f6a48a50640 1 -- 172.21.15.37:0/3912595396 learned_addr learned my addr 172.21.15.37:0/3912595396 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:39.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.241+0000 7f6a42ffd640 1 -- 172.21.15.37:0/3912595396 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 369598386 0 0) 0x7f6a440af940 con 0x7f6a4414be90 2024-03-25T18:17:39.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.241+0000 7f6a42ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:39.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.241+0000 7f6a42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:39.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.241+0000 7f6a42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:39.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.241+0000 7f6a42ffd640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:17:39.242 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.241+0000 7f6a42ffd640 1 -- 172.21.15.37:0/3912595396 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a2c003520 con 0x7f6a4414be90 2024-03-25T18:17:39.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 1 -- 172.21.15.37:0/3912595396 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 488536581 0 0) 0x7f6a2c003520 con 0x7f6a4414be90 2024-03-25T18:17:39.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 1 -- 172.21.15.37:0/3912595396 >> v1:172.21.15.37:6790/0 conn(0x7f6a440b4d60 legacy=0x7f6a4413f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:17:39.255 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 1 -- 172.21.15.37:0/3912595396 >> v1:172.21.15.117:6789/0 conn(0x7f6a440b1730 legacy=0x7f6a440b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 1 monclient: found mon.noname-a 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 1 -- 172.21.15.37:0/3912595396 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a440b1030 con 0x7f6a4414be90 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 1 -- 172.21.15.37:0/3912595396 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6a3c001a10 con 0x7f6a4414be90 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a4a4b3640 5 monclient: authenticate success, global_id 4677 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a4a4b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a42ffd640 10 monclient: dump: 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:39.256 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:39.257 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.257 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.257 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:39.257 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:39.257 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:39.257 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:39.257 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:39.257 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:39.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.254+0000 7f6a4a4b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.270+0000 7f6a42ffd640 1 -- 172.21.15.37:0/3912595396 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a3c001be0 con 0x7f6a4414be90 2024-03-25T18:17:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.270+0000 7f6a42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.270+0000 7f6a42ffd640 1 -- 172.21.15.37:0/3912595396 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6a3c002030 con 0x7f6a4414be90 2024-03-25T18:17:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.270+0000 7f6a4a4b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.270+0000 7f6a4a4b3640 4 set_mon_vals no callback set 2024-03-25T18:17:39.271 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.270+0000 7f6a42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.270+0000 7f6a42ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.270+0000 7f6a42ffd640 10 monclient: dump: 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:39.272 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.271+0000 7f6a4a4b3640 10 monclient: shutdown 2024-03-25T18:17:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.271+0000 7f6a4a4b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.271+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3912595396 >> v1:172.21.15.37:6789/0 conn(0x7f6a4414be90 legacy=0x7f6a4414e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:39.274 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.271+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3912595396 shutdown_connections 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.271+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3912595396 >> 172.21.15.37:0/3912595396 conn(0x7f6a440617a0 msgr2=0x7f6a44061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.271+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3912595396 shutdown_connections 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.271+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3912595396 wait complete. 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.271+0000 7f6a4a4b3640 10 monclient: build_initial_monmap 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.272+0000 7f6a4a4b3640 10 monclient: monmap: 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:39.275 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.272+0000 7f6a4a4b3640 1 librados: starting msgr at 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.272+0000 7f6a4a4b3640 1 librados: starting objecter 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.272+0000 7f6a4a4b3640 1 Processor -- start 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.272+0000 7f6a4a4b3640 1 -- start start 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.272+0000 7f6a4a4b3640 1 librados: setting wanted keys 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.272+0000 7f6a4a4b3640 1 librados: calling monclient init 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.272+0000 7f6a4a4b3640 10 monclient: init 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.273+0000 7f6a4a4b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.273+0000 7f6a4a4b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.273+0000 7f6a4a4b3640 10 monclient(hunting): picked mon.noname-b con 0x7f6a440b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.273+0000 7f6a4a4b3640 10 monclient(hunting): picked mon.noname-a con 0x7f6a440b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.273+0000 7f6a4a4b3640 10 monclient(hunting): picked mon.noname-c con 0x7f6a4414be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.273+0000 7f6a4a4b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a441d7590 con 0x7f6a440b4d60 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.273+0000 7f6a4a4b3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a44138840 con 0x7f6a440b1730 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.273+0000 7f6a4a4b3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a440aa6d0 con 0x7f6a4414be90 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.273+0000 7f6a4a4b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:39.276 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.273+0000 7f6a4a4b3640 10 monclient(hunting): authenticate will time out at 7414.525689s 2024-03-25T18:17:39.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.285+0000 7f6a48a50640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f6a4414be90 0x7f6a44138130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:49612/0 (socket says 172.21.15.37:49612) 2024-03-25T18:17:39.286 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.285+0000 7f6a48a50640 1 -- 172.21.15.37:0/3273079406 learned_addr learned my addr 172.21.15.37:0/3273079406 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:39.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.286+0000 7f6a40ff9640 10 client.?.objecter ms_handle_connect 0x7f6a4414be90 2024-03-25T18:17:39.287 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.286+0000 7f6a40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:39.293 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.293+0000 7f6a40ff9640 10 client.?.objecter ms_handle_connect 0x7f6a440b1730 2024-03-25T18:17:39.294 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.293+0000 7f6a40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:39.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.298+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3015288142 0 0) 0x7f6a440aa6d0 con 0x7f6a4414be90 2024-03-25T18:17:39.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.298+0000 7f6a40ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:39.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.298+0000 7f6a40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:39.299 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.298+0000 7f6a40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:39.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.298+0000 7f6a40ff9640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:17:39.300 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.298+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a30004eb0 con 0x7f6a4414be90 2024-03-25T18:17:39.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.302+0000 7f6a40ff9640 10 client.?.objecter ms_handle_connect 0x7f6a440b4d60 2024-03-25T18:17:39.303 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.302+0000 7f6a40ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:39.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.303+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4006714537 0 0) 0x7f6a44138840 con 0x7f6a440b1730 2024-03-25T18:17:39.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.304+0000 7f6a40ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:39.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.304+0000 7f6a40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:39.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.304+0000 7f6a40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:39.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.304+0000 7f6a40ff9640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:17:39.305 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.304+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a440aa6d0 con 0x7f6a440b1730 2024-03-25T18:17:39.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.319+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2562118690 0 0) 0x7f6a30004eb0 con 0x7f6a4414be90 2024-03-25T18:17:39.322 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.319+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a44138840 con 0x7f6a4414be90 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.319+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6a3c001f30 con 0x7f6a4414be90 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.319+0000 7f6a40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.319+0000 7f6a40ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.319+0000 7f6a40ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.319+0000 7f6a40ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:39.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.320+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 596090626 0 0) 0x7f6a440aa6d0 con 0x7f6a440b1730 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.320+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a30004eb0 con 0x7f6a440b1730 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.320+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6a34001810 con 0x7f6a440b1730 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.320+0000 7f6a40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.320+0000 7f6a40ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.320+0000 7f6a40ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.321+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2554940636 0 0) 0x7f6a441d7590 con 0x7f6a440b4d60 2024-03-25T18:17:39.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.321+0000 7f6a40ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:39.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.321+0000 7f6a40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:39.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.321+0000 7f6a40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:39.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.321+0000 7f6a40ff9640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:17:39.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.321+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a440aa6d0 con 0x7f6a440b4d60 2024-03-25T18:17:39.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.334+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 723844309 0 0) 0x7f6a440aa6d0 con 0x7f6a440b4d60 2024-03-25T18:17:39.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.334+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a441d7590 con 0x7f6a440b4d60 2024-03-25T18:17:39.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.334+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6a380017d0 con 0x7f6a440b4d60 2024-03-25T18:17:39.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.334+0000 7f6a40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:39.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.334+0000 7f6a40ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:39.336 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.334+0000 7f6a40ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:39.336 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:39.336 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:39.336 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.337 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.337 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:39.337 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:39.337 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:39.337 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:39.337 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:39.337 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:39.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1468568391 0 0) 0x7f6a44138840 con 0x7f6a4414be90 2024-03-25T18:17:39.338 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 >> v1:172.21.15.117:6789/0 conn(0x7f6a440b1730 legacy=0x7f6a441d7a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 >> v1:172.21.15.37:6789/0 conn(0x7f6a440b4d60 legacy=0x7f6a44137a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a40ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a40ff9640 1 monclient: found mon.c 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a40ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a40ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a440aa9b0 con 0x7f6a4414be90 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a40ff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a40ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a4a4b3640 5 monclient: authenticate success, global_id 4664 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a4a4b3640 10 monclient: _renew_subs 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a4a4b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:39.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3273079406 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a44138c40 con 0x7f6a4414be90 2024-03-25T18:17:39.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a4a4b3640 10 client.4664.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:39.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a4a4b3640 10 monclient: _renew_subs 2024-03-25T18:17:39.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a4a4b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:39.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3273079406 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a4413b2c0 con 0x7f6a4414be90 2024-03-25T18:17:39.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.337+0000 7f6a4a4b3640 1 librados: init done 2024-03-25T18:17:39.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.338+0000 7f6a4a4b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:39.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.338+0000 7f6a4a4b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:39.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.338+0000 7f6a4a4b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:39.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.338+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3273079406 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a4413b5c0 con 0x7f6a4414be90 2024-03-25T18:17:39.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.714+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a3c004e10 con 0x7f6a4414be90 2024-03-25T18:17:39.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.714+0000 7f6a40ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:39.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.714+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6a3c0015f0 con 0x7f6a4414be90 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.714+0000 7f6a40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.714+0000 7f6a40ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.714+0000 7f6a40ff9640 10 monclient: dump: 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:39.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.714+0000 7f6a427fc640 4 set_mon_vals no callback set 2024-03-25T18:17:39.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.715+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f6a3c01af90 con 0x7f6a4414be90 2024-03-25T18:17:39.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.716+0000 7f6a43fff640 10 monclient: get_auth_request con 0x7f6a3006e0f0 auth_method 0 2024-03-25T18:17:39.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.716+0000 7f6a40ff9640 10 client.4664.objecter ms_handle_connect 0x7f6a3006e0f0 2024-03-25T18:17:39.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.721+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 2144329973 0 0) 0x7f6a3c054b00 con 0x7f6a4414be90 2024-03-25T18:17:39.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.721+0000 7f6a40ff9640 10 client.4664.objecter ms_dispatch 0x7f6a44000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:39.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.721+0000 7f6a40ff9640 3 client.4664.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:39.722 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.721+0000 7f6a40ff9640 3 client.4664.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:39.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.721+0000 7f6a40ff9640 20 client.4664.objecter dump_active .. 0 homeless 2024-03-25T18:17:39.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.721+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a3c050610 con 0x7f6a4414be90 2024-03-25T18:17:39.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.721+0000 7f6a40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:39.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.721+0000 7f6a40ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:39.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.984+0000 7f6a4a4b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:39.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.985+0000 7f6a4a4b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:39.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.985+0000 7f6a4a4b3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:39.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:39.985+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3273079406 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a440afbd0 con 0x7f6a4414be90 2024-03-25T18:17:40.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.045+0000 7f6a40ff9640 1 -- 172.21.15.37:0/3273079406 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 223660573) 0x7f6a3c052540 con 0x7f6a4414be90 2024-03-25T18:17:40.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.045+0000 7f6a40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:40.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.045+0000 7f6a40ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:40.047 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:17:40.047 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":10,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:07:47.505088+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:07:47.505088+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:40.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 10 librados: watch_flush enter 2024-03-25T18:17:40.049 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 10 librados: watch_flush exit 2024-03-25T18:17:40.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 20 client.4664.objecter shutdown clearing up homeless session... 2024-03-25T18:17:40.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 10 client.4664.objecter successfully canceled tick 2024-03-25T18:17:40.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3273079406 >> v1:172.21.15.117:6800/3703993921 conn(0x7f6a3006e0f0 legacy=0x7f6a300705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:40.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 10 monclient: shutdown 2024-03-25T18:17:40.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:40.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3273079406 >> v1:172.21.15.37:6790/0 conn(0x7f6a4414be90 legacy=0x7f6a44138130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:40.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3273079406 shutdown_connections 2024-03-25T18:17:40.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3273079406 >> 172.21.15.37:0/3273079406 conn(0x7f6a440617a0 msgr2=0x7f6a44145030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:40.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3273079406 shutdown_connections 2024-03-25T18:17:40.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 1 -- 172.21.15.37:0/3273079406 wait complete. 2024-03-25T18:17:40.050 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:40.048+0000 7f6a4a4b3640 1 librados: shutdown 2024-03-25T18:17:40.062 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:46.868 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:46.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.974+0000 7f36bda55640 10 monclient: get_monmap_and_config 2024-03-25T18:17:46.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.974+0000 7f36bda55640 10 monclient: build_initial_monmap 2024-03-25T18:17:46.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.974+0000 7f36bda55640 10 monclient: monmap: 2024-03-25T18:17:46.976 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:46.976 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:46.976 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:17:46.975352+0000 2024-03-25T18:17:46.976 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:17:46.975352+0000 2024-03-25T18:17:46.976 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:46.976 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:46.976 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:46.976 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:17:46.976 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:17:46.976 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:46.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.975+0000 7f36bda55640 1 Processor -- start 2024-03-25T18:17:46.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.975+0000 7f36bda55640 1 -- start start 2024-03-25T18:17:46.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.975+0000 7f36bda55640 10 monclient: init 2024-03-25T18:17:46.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.976+0000 7f36bda55640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:46.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.976+0000 7f36bda55640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:46.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.976+0000 7f36bda55640 10 monclient(hunting): picked mon.noname-a con 0x7f36b80a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:46.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.976+0000 7f36bda55640 10 monclient(hunting): picked mon.noname-b con 0x7f36b813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:46.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.976+0000 7f36bda55640 10 monclient(hunting): picked mon.noname-c con 0x7f36b81430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:46.977 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.976+0000 7f36bda55640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b8142520 con 0x7f36b80a98f0 2024-03-25T18:17:46.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.976+0000 7f36bda55640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b80b28a0 con 0x7f36b81430c0 2024-03-25T18:17:46.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.976+0000 7f36bda55640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b80b2a30 con 0x7f36b813f710 2024-03-25T18:17:46.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.976+0000 7f36bda55640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:46.978 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.976+0000 7f36bda55640 10 monclient(hunting): authenticate will time out at 7422.228589s 2024-03-25T18:17:46.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.990+0000 7f36b77fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f36b81430c0 0x7f36b814dc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:60850/0 (socket says 172.21.15.37:60850) 2024-03-25T18:17:46.991 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.990+0000 7f36b77fe640 1 -- 172.21.15.37:0/2209978284 learned_addr learned my addr 172.21.15.37:0/2209978284 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:46.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.998+0000 7f36b5ffb640 1 -- 172.21.15.37:0/2209978284 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370697931 0 0) 0x7f36b80b28a0 con 0x7f36b81430c0 2024-03-25T18:17:46.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.998+0000 7f36b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:46.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.998+0000 7f36b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:46.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.998+0000 7f36b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:46.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.998+0000 7f36b5ffb640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:17:46.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.998+0000 7f36b5ffb640 1 -- 172.21.15.37:0/2209978284 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a0003520 con 0x7f36b81430c0 2024-03-25T18:17:47.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.999+0000 7f36b5ffb640 1 -- 172.21.15.37:0/2209978284 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1127646073 0 0) 0x7f36b8142520 con 0x7f36b80a98f0 2024-03-25T18:17:47.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.999+0000 7f36b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:47.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.999+0000 7f36b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:47.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.999+0000 7f36b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:47.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.999+0000 7f36b5ffb640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:17:47.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:46.999+0000 7f36b5ffb640 1 -- 172.21.15.37:0/2209978284 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b80b28a0 con 0x7f36b80a98f0 2024-03-25T18:17:47.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 1 -- 172.21.15.37:0/2209978284 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 60561483 0 0) 0x7f36b80b28a0 con 0x7f36b80a98f0 2024-03-25T18:17:47.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 1 -- 172.21.15.37:0/2209978284 >> v1:172.21.15.37:6790/0 conn(0x7f36b813f710 legacy=0x7f36b81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:47.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 1 -- 172.21.15.37:0/2209978284 >> v1:172.21.15.117:6789/0 conn(0x7f36b81430c0 legacy=0x7f36b814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:47.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 1 monclient: found mon.noname-a 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 1 -- 172.21.15.37:0/2209978284 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b80b44c0 con 0x7f36b80a98f0 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 1 -- 172.21.15.37:0/2209978284 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f36a8001670 con 0x7f36b80a98f0 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36b5ffb640 10 monclient: dump: 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:47.008 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:47.009 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:47.009 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:47.009 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:47.009 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:47.009 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:47.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36bda55640 5 monclient: authenticate success, global_id 4683 2024-03-25T18:17:47.009 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.006+0000 7f36bda55640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:47.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.014+0000 7f36b5ffb640 1 -- 172.21.15.37:0/2209978284 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36a80017d0 con 0x7f36b80a98f0 2024-03-25T18:17:47.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.014+0000 7f36b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:47.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.014+0000 7f36b5ffb640 1 -- 172.21.15.37:0/2209978284 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f36a8001b60 con 0x7f36b80a98f0 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.014+0000 7f36bda55640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.014+0000 7f36bda55640 4 set_mon_vals no callback set 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.014+0000 7f36b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.014+0000 7f36b5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.014+0000 7f36b5ffb640 10 monclient: dump: 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:47.016 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:47.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.015+0000 7f36bda55640 10 monclient: shutdown 2024-03-25T18:17:47.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.015+0000 7f36bda55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:47.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.015+0000 7f36bda55640 1 -- 172.21.15.37:0/2209978284 >> v1:172.21.15.37:6789/0 conn(0x7f36b80a98f0 legacy=0x7f36b80a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:47.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.015+0000 7f36bda55640 1 -- 172.21.15.37:0/2209978284 shutdown_connections 2024-03-25T18:17:47.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.015+0000 7f36bda55640 1 -- 172.21.15.37:0/2209978284 >> 172.21.15.37:0/2209978284 conn(0x7f36b80b7ea0 msgr2=0x7f36b80b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:47.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.015+0000 7f36bda55640 1 -- 172.21.15.37:0/2209978284 shutdown_connections 2024-03-25T18:17:47.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.015+0000 7f36bda55640 1 -- 172.21.15.37:0/2209978284 wait complete. 2024-03-25T18:17:47.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.016+0000 7f36bda55640 10 monclient: build_initial_monmap 2024-03-25T18:17:47.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.016+0000 7f36bda55640 10 monclient: monmap: 2024-03-25T18:17:47.018 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.016+0000 7f36bda55640 1 librados: starting msgr at 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.016+0000 7f36bda55640 1 librados: starting objecter 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.016+0000 7f36bda55640 1 Processor -- start 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.016+0000 7f36bda55640 1 -- start start 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.016+0000 7f36bda55640 1 librados: setting wanted keys 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.016+0000 7f36bda55640 1 librados: calling monclient init 2024-03-25T18:17:47.019 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.016+0000 7f36bda55640 10 monclient: init 2024-03-25T18:17:47.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.017+0000 7f36bda55640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:47.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.017+0000 7f36bda55640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:47.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.017+0000 7f36bda55640 10 monclient(hunting): picked mon.noname-b con 0x7f36b80a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:47.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.017+0000 7f36bda55640 10 monclient(hunting): picked mon.noname-c con 0x7f36b813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:47.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.018+0000 7f36bda55640 10 monclient(hunting): picked mon.noname-a con 0x7f36b81430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:47.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.018+0000 7f36bda55640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b813bf90 con 0x7f36b81430c0 2024-03-25T18:17:47.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.018+0000 7f36bda55640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b813b060 con 0x7f36b80a98f0 2024-03-25T18:17:47.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.018+0000 7f36bda55640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36b81dc070 con 0x7f36b813f710 2024-03-25T18:17:47.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.018+0000 7f36bda55640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:47.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.018+0000 7f36bda55640 10 monclient(hunting): authenticate will time out at 7422.270418s 2024-03-25T18:17:47.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.025+0000 7f36b67fc640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f36b813f710 0x7f36b81db250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:54214/0 (socket says 172.21.15.37:54214) 2024-03-25T18:17:47.026 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.025+0000 7f36b67fc640 1 -- 172.21.15.37:0/1701268153 learned_addr learned my addr 172.21.15.37:0/1701268153 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:47.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.026+0000 7f369ffff640 10 client.?.objecter ms_handle_connect 0x7f36b813f710 2024-03-25T18:17:47.027 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.026+0000 7f369ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:47.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.029+0000 7f369ffff640 10 client.?.objecter ms_handle_connect 0x7f36b80a98f0 2024-03-25T18:17:47.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.029+0000 7f369ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:47.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.033+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2747249442 0 0) 0x7f36b81dc070 con 0x7f36b813f710 2024-03-25T18:17:47.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.033+0000 7f369ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:47.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.033+0000 7f369ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:47.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.033+0000 7f369ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:47.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.033+0000 7f369ffff640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:17:47.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.033+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36a4004eb0 con 0x7f36b813f710 2024-03-25T18:17:47.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.036+0000 7f369ffff640 10 client.?.objecter ms_handle_connect 0x7f36b81430c0 2024-03-25T18:17:47.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.036+0000 7f369ffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:47.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.037+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4222243645 0 0) 0x7f36b813b060 con 0x7f36b80a98f0 2024-03-25T18:17:47.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.037+0000 7f369ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:47.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.037+0000 7f369ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:47.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.037+0000 7f369ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:47.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.037+0000 7f369ffff640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:17:47.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.037+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b81dc070 con 0x7f36b80a98f0 2024-03-25T18:17:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657761208 0 0) 0x7f36b813bf90 con 0x7f36b81430c0 2024-03-25T18:17:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:47.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 10 monclient(hunting): my global_id is 4686 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f36b813b060 con 0x7f36b81430c0 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 933635835 0 0) 0x7f36a4004eb0 con 0x7f36b813f710 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36b813bf90 con 0x7f36b813f710 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f36ac001a30 con 0x7f36b813f710 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 10 monclient(hunting): dump: 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:47.052 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:47.053 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:47.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.050+0000 7f369ffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:47.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.057+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3416865097 0 0) 0x7f36b81dc070 con 0x7f36b80a98f0 2024-03-25T18:17:47.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.057+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f36a4004eb0 con 0x7f36b80a98f0 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.057+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f36a8001ae0 con 0x7f36b80a98f0 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.057+0000 7f369ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.057+0000 7f369ffff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.057+0000 7f369ffff640 10 monclient(hunting): dump: 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:47.059 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:47.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3836655467 0 0) 0x7f36b813bf90 con 0x7f36b813f710 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 >> v1:172.21.15.117:6789/0 conn(0x7f36b80a98f0 legacy=0x7f36b813c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 >> v1:172.21.15.37:6789/0 conn(0x7f36b81430c0 legacy=0x7f36b81db960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f369ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f369ffff640 1 monclient: found mon.c 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f369ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f369ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36b813b2d0 con 0x7f36b813f710 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f369ffff640 10 monclient: _finish_auth 0 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f369ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f369ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f36bda55640 5 monclient: authenticate success, global_id 4670 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f36bda55640 10 monclient: _renew_subs 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f36bda55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f36bda55640 1 -- 172.21.15.37:0/1701268153 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36b813b5b0 con 0x7f36b813f710 2024-03-25T18:17:47.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f36bda55640 10 client.4670.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:47.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f36bda55640 10 monclient: _renew_subs 2024-03-25T18:17:47.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f36bda55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:47.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f36bda55640 1 -- 172.21.15.37:0/1701268153 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f36b80b7660 con 0x7f36b813f710 2024-03-25T18:17:47.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.060+0000 7f36bda55640 1 librados: init done 2024-03-25T18:17:47.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.061+0000 7f36bda55640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:47.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.061+0000 7f36bda55640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:47.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.061+0000 7f36bda55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:47.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.061+0000 7f36bda55640 1 -- 172.21.15.37:0/1701268153 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36b80b3ef0 con 0x7f36b813f710 2024-03-25T18:17:47.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.388+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f36ac001c70 con 0x7f36b813f710 2024-03-25T18:17:47.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.388+0000 7f369ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:47.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.388+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f36ac003a60 con 0x7f36b813f710 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.388+0000 7f369ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.388+0000 7f369ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.388+0000 7f369ffff640 10 monclient: dump: 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:47.390 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:47.391 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:47.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.388+0000 7f36b57fa640 4 set_mon_vals no callback set 2024-03-25T18:17:47.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.389+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f36ac003c80 con 0x7f36b813f710 2024-03-25T18:17:47.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.389+0000 7f36b6ffd640 10 monclient: get_auth_request con 0x7f36a406df70 auth_method 0 2024-03-25T18:17:47.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.390+0000 7f369ffff640 10 client.4670.objecter ms_handle_connect 0x7f36a406df70 2024-03-25T18:17:47.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.396+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7f36ac0548a0 con 0x7f36b813f710 2024-03-25T18:17:47.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.396+0000 7f369ffff640 10 client.4670.objecter ms_dispatch 0x7f36b8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:17:47.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.396+0000 7f369ffff640 3 client.4670.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:17:47.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.396+0000 7f369ffff640 3 client.4670.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:17:47.398 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.396+0000 7f369ffff640 20 client.4670.objecter dump_active .. 0 homeless 2024-03-25T18:17:47.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.397+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f36ac0503b0 con 0x7f36b813f710 2024-03-25T18:17:47.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.397+0000 7f369ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:47.399 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.397+0000 7f369ffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:47.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.624+0000 7f36bda55640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:47.625 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.624+0000 7f36bda55640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:47.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.624+0000 7f36bda55640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:47.626 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.624+0000 7f36bda55640 1 -- 172.21.15.37:0/1701268153 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f36b80627b0 con 0x7f36b813f710 2024-03-25T18:17:47.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.696+0000 7f369ffff640 1 -- 172.21.15.37:0/1701268153 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7f36ac0522e0 con 0x7f36b813f710 2024-03-25T18:17:47.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.696+0000 7f369ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:47.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.696+0000 7f369ffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:47.698 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:17:47.698 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:47.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.698+0000 7f36bda55640 10 librados: watch_flush enter 2024-03-25T18:17:47.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.698+0000 7f36bda55640 10 librados: watch_flush exit 2024-03-25T18:17:47.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.698+0000 7f36bda55640 20 client.4670.objecter shutdown clearing up homeless session... 2024-03-25T18:17:47.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.698+0000 7f36bda55640 10 client.4670.objecter successfully canceled tick 2024-03-25T18:17:47.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.698+0000 7f36bda55640 1 -- 172.21.15.37:0/1701268153 >> v1:172.21.15.117:6800/3703993921 conn(0x7f36a406df70 legacy=0x7f36a4070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:47.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.698+0000 7f36bda55640 10 monclient: shutdown 2024-03-25T18:17:47.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.698+0000 7f36bda55640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:47.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.698+0000 7f36bda55640 1 -- 172.21.15.37:0/1701268153 >> v1:172.21.15.37:6790/0 conn(0x7f36b813f710 legacy=0x7f36b81db250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:47.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.699+0000 7f36bda55640 1 -- 172.21.15.37:0/1701268153 shutdown_connections 2024-03-25T18:17:47.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.699+0000 7f36bda55640 1 -- 172.21.15.37:0/1701268153 >> 172.21.15.37:0/1701268153 conn(0x7f36b80b7ea0 msgr2=0x7f36b8074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:47.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.699+0000 7f36bda55640 1 -- 172.21.15.37:0/1701268153 shutdown_connections 2024-03-25T18:17:47.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.699+0000 7f36bda55640 1 -- 172.21.15.37:0/1701268153 wait complete. 2024-03-25T18:17:47.702 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:47.699+0000 7f36bda55640 1 librados: shutdown 2024-03-25T18:17:47.712 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:54.518 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:54.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.632+0000 7f18f9286640 10 monclient: get_monmap_and_config 2024-03-25T18:17:54.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.632+0000 7f18f9286640 10 monclient: build_initial_monmap 2024-03-25T18:17:54.633 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.632+0000 7f18f9286640 10 monclient: monmap: 2024-03-25T18:17:54.634 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:54.634 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:54.634 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:17:54.633240+0000 2024-03-25T18:17:54.634 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:17:54.633240+0000 2024-03-25T18:17:54.634 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:54.634 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:54.634 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:54.634 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:17:54.634 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:17:54.634 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 1 Processor -- start 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 1 -- start start 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 10 monclient: init 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 10 monclient(hunting): picked mon.noname-c con 0x7f18f40a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 10 monclient(hunting): picked mon.noname-a con 0x7f18f413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 10 monclient(hunting): picked mon.noname-b con 0x7f18f41430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f4142520 con 0x7f18f413f710 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f40b28a0 con 0x7f18f40a98f0 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f40b2a30 con 0x7f18f41430c0 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:54.636 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.633+0000 7f18f9286640 10 monclient(hunting): authenticate will time out at 7429.886205s 2024-03-25T18:17:54.643 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.642+0000 7f18f2ffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f18f40a98f0 0x7f18f40a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:47988/0 (socket says 172.21.15.37:47988) 2024-03-25T18:17:54.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.642+0000 7f18f2ffd640 1 -- 172.21.15.37:0/4060933661 learned_addr learned my addr 172.21.15.37:0/4060933661 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:54.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.656+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883585666 0 0) 0x7f18f40b28a0 con 0x7f18f40a98f0 2024-03-25T18:17:54.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.656+0000 7f18f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:54.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.656+0000 7f18f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:54.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.656+0000 7f18f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:54.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.656+0000 7f18f1ffb640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:17:54.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.656+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18dc003520 con 0x7f18f40a98f0 2024-03-25T18:17:54.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.663+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1353817033 0 0) 0x7f18f4142520 con 0x7f18f413f710 2024-03-25T18:17:54.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.663+0000 7f18f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:54.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.663+0000 7f18f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:54.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.663+0000 7f18f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:54.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.663+0000 7f18f1ffb640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:17:54.664 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.663+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f40b28a0 con 0x7f18f413f710 2024-03-25T18:17:54.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.669+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223546620 0 0) 0x7f18f40b2a30 con 0x7f18f41430c0 2024-03-25T18:17:54.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.669+0000 7f18f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:17:54.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.669+0000 7f18f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:54.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.669+0000 7f18f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:54.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.669+0000 7f18f1ffb640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:17:54.670 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.669+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18f4142520 con 0x7f18f41430c0 2024-03-25T18:17:54.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3105057043 0 0) 0x7f18dc003520 con 0x7f18f40a98f0 2024-03-25T18:17:54.672 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 >> v1:172.21.15.37:6790/0 conn(0x7f18f41430c0 legacy=0x7f18f414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 >> v1:172.21.15.37:6789/0 conn(0x7f18f413f710 legacy=0x7f18f41429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 1 monclient: found mon.noname-c 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f40b44c0 con 0x7f18f40a98f0 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f18e4001690 con 0x7f18f40a98f0 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f9286640 5 monclient: authenticate success, global_id 4675 2024-03-25T18:17:54.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f9286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:54.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:54.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:54.674 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f1ffb640 10 monclient: dump: 2024-03-25T18:17:54.674 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:54.674 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:54.674 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:54.674 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:54.674 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:54.674 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:54.674 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:54.675 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:54.675 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:54.675 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:54.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.671+0000 7f18f9286640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:54.682 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.681+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18e40017f0 con 0x7f18f40a98f0 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.681+0000 7f18f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.681+0000 7f18f1ffb640 1 -- 172.21.15.37:0/4060933661 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f18e4001c00 con 0x7f18f40a98f0 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.681+0000 7f18f9286640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.681+0000 7f18f9286640 4 set_mon_vals no callback set 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.681+0000 7f18f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.681+0000 7f18f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.681+0000 7f18f1ffb640 10 monclient: dump: 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:54.683 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:54.684 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:54.684 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:54.684 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:54.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.682+0000 7f18f9286640 10 monclient: shutdown 2024-03-25T18:17:54.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.682+0000 7f18f9286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:54.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.682+0000 7f18f9286640 1 -- 172.21.15.37:0/4060933661 >> v1:172.21.15.117:6789/0 conn(0x7f18f40a98f0 legacy=0x7f18f40a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:54.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.682+0000 7f18f9286640 1 -- 172.21.15.37:0/4060933661 shutdown_connections 2024-03-25T18:17:54.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.682+0000 7f18f9286640 1 -- 172.21.15.37:0/4060933661 >> 172.21.15.37:0/4060933661 conn(0x7f18f40b7ea0 msgr2=0x7f18f40b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:54.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.682+0000 7f18f9286640 1 -- 172.21.15.37:0/4060933661 shutdown_connections 2024-03-25T18:17:54.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.682+0000 7f18f9286640 1 -- 172.21.15.37:0/4060933661 wait complete. 2024-03-25T18:17:54.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.683+0000 7f18f9286640 10 monclient: build_initial_monmap 2024-03-25T18:17:54.686 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.683+0000 7f18f9286640 10 monclient: monmap: 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.683+0000 7f18f9286640 1 librados: starting msgr at 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.683+0000 7f18f9286640 1 librados: starting objecter 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.683+0000 7f18f9286640 1 Processor -- start 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 1 -- start start 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 1 librados: setting wanted keys 2024-03-25T18:17:54.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 1 librados: calling monclient init 2024-03-25T18:17:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 10 monclient: init 2024-03-25T18:17:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 10 monclient(hunting): picked mon.noname-c con 0x7f18f40a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:17:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 10 monclient(hunting): picked mon.noname-a con 0x7f18f413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:17:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 10 monclient(hunting): picked mon.noname-b con 0x7f18f41430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:17:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f4138a50 con 0x7f18f413f710 2024-03-25T18:17:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f40ab9e0 con 0x7f18f41430c0 2024-03-25T18:17:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18f41d7880 con 0x7f18f40a98f0 2024-03-25T18:17:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:54.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.684+0000 7f18f9286640 10 monclient(hunting): authenticate will time out at 7429.936928s 2024-03-25T18:17:54.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.689+0000 7f18f2ffd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f18f40a98f0 0x7f18f4138340 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:42592/0 (socket says 172.21.15.37:42592) 2024-03-25T18:17:54.690 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.689+0000 7f18f2ffd640 1 -- 172.21.15.37:0/52864408 learned_addr learned my addr 172.21.15.37:0/52864408 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:17:54.691 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.690+0000 7f18db7fe640 10 client.?.objecter ms_handle_connect 0x7f18f40a98f0 2024-03-25T18:17:54.692 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.690+0000 7f18db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:54.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.695+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2199367211 0 0) 0x7f18f41d7880 con 0x7f18f40a98f0 2024-03-25T18:17:54.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.696+0000 7f18db7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:17:54.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.696+0000 7f18db7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:54.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.696+0000 7f18db7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:54.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.696+0000 7f18db7fe640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:17:54.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.696+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18e0004eb0 con 0x7f18f40a98f0 2024-03-25T18:17:54.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.697+0000 7f18db7fe640 10 client.?.objecter ms_handle_connect 0x7f18f41430c0 2024-03-25T18:17:54.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.697+0000 7f18db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:54.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.705+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3112012672 0 0) 0x7f18e0004eb0 con 0x7f18f40a98f0 2024-03-25T18:17:54.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.705+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f18f41d7880 con 0x7f18f40a98f0 2024-03-25T18:17:54.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.705+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f18e4001bc0 con 0x7f18f40a98f0 2024-03-25T18:17:54.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.705+0000 7f18db7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:54.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.705+0000 7f18db7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:54.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.705+0000 7f18db7fe640 10 monclient(hunting): dump: 2024-03-25T18:17:54.706 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:54.706 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:54.706 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:54.706 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:54.707 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:54.707 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:54.707 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:54.707 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:54.707 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:54.707 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:54.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.705+0000 7f18db7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:54.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.707+0000 7f18db7fe640 10 client.?.objecter ms_handle_connect 0x7f18f413f710 2024-03-25T18:17:54.708 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.707+0000 7f18db7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:54.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 370824488 0 0) 0x7f18f41d7880 con 0x7f18f40a98f0 2024-03-25T18:17:54.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 >> v1:172.21.15.117:6789/0 conn(0x7f18f41430c0 legacy=0x7f18f41e1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:54.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 >> v1:172.21.15.37:6789/0 conn(0x7f18f413f710 legacy=0x7f18f41d7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:54.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18db7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:17:54.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18db7fe640 1 monclient: found mon.c 2024-03-25T18:17:54.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18db7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:54.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18db7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:54.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18f41d84b0 con 0x7f18f40a98f0 2024-03-25T18:17:54.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18db7fe640 10 monclient: _finish_auth 0 2024-03-25T18:17:54.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18db7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:17:54.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18db7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:54.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18f9286640 5 monclient: authenticate success, global_id 4676 2024-03-25T18:17:54.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18f9286640 10 monclient: _renew_subs 2024-03-25T18:17:54.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18f9286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:54.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18f9286640 1 -- 172.21.15.37:0/52864408 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18f41d51f0 con 0x7f18f40a98f0 2024-03-25T18:17:54.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18f9286640 10 client.4676.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:54.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18f9286640 10 monclient: _renew_subs 2024-03-25T18:17:54.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18f9286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:54.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18f9286640 1 -- 172.21.15.37:0/52864408 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f18f4138e60 con 0x7f18f40a98f0 2024-03-25T18:17:54.716 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.713+0000 7f18f9286640 1 librados: init done 2024-03-25T18:17:54.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.715+0000 7f18f9286640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:54.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.715+0000 7f18f9286640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:54.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.715+0000 7f18f9286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:54.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:54.715+0000 7f18f9286640 1 -- 172.21.15.37:0/52864408 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18f41d76e0 con 0x7f18f40a98f0 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.039+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18e4004a90 con 0x7f18f40a98f0 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.039+0000 7f18db7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.039+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f18e4004d00 con 0x7f18f40a98f0 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.039+0000 7f18db7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.039+0000 7f18db7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.039+0000 7f18db7fe640 10 monclient: dump: 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:17:55.041 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:17:55.042 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:17:55.042 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:17:55.042 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:17:55.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.039+0000 7f18f17fa640 4 set_mon_vals no callback set 2024-03-25T18:17:55.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.040+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f18e4004f20 con 0x7f18f40a98f0 2024-03-25T18:17:55.042 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.041+0000 7f18f27fc640 10 monclient: get_auth_request con 0x7f18e006dd00 auth_method 0 2024-03-25T18:17:55.043 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.041+0000 7f18db7fe640 10 client.4676.objecter ms_handle_connect 0x7f18e006dd00 2024-03-25T18:17:55.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.046+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7f18e40547d0 con 0x7f18f40a98f0 2024-03-25T18:17:55.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.046+0000 7f18db7fe640 10 client.4676.objecter ms_dispatch 0x7f18f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:17:55.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.046+0000 7f18db7fe640 3 client.4676.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:17:55.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.046+0000 7f18db7fe640 3 client.4676.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:17:55.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.046+0000 7f18db7fe640 20 client.4676.objecter dump_active .. 0 homeless 2024-03-25T18:17:55.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.046+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f18e40502e0 con 0x7f18f40a98f0 2024-03-25T18:17:55.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.046+0000 7f18db7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:55.048 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.046+0000 7f18db7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:55.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.281+0000 7f18f9286640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:55.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.282+0000 7f18f9286640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:55.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.282+0000 7f18f9286640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:17:55.283 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.282+0000 7f18f9286640 1 -- 172.21.15.37:0/52864408 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18f40627b0 con 0x7f18f40a98f0 2024-03-25T18:17:55.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.383+0000 7f18db7fe640 1 -- 172.21.15.37:0/52864408 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7f18e4052210 con 0x7f18f40a98f0 2024-03-25T18:17:55.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.383+0000 7f18db7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:55.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.383+0000 7f18db7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:55.385 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:17:55.385 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:55.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 10 librados: watch_flush enter 2024-03-25T18:17:55.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 10 librados: watch_flush exit 2024-03-25T18:17:55.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 20 client.4676.objecter shutdown clearing up homeless session... 2024-03-25T18:17:55.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 10 client.4676.objecter successfully canceled tick 2024-03-25T18:17:55.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 1 -- 172.21.15.37:0/52864408 >> v1:172.21.15.117:6800/3703993921 conn(0x7f18e006dd00 legacy=0x7f18e00701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:55.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 10 monclient: shutdown 2024-03-25T18:17:55.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:55.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 1 -- 172.21.15.37:0/52864408 >> v1:172.21.15.37:6790/0 conn(0x7f18f40a98f0 legacy=0x7f18f4138340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:55.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 1 -- 172.21.15.37:0/52864408 shutdown_connections 2024-03-25T18:17:55.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 1 -- 172.21.15.37:0/52864408 >> 172.21.15.37:0/52864408 conn(0x7f18f40b7ea0 msgr2=0x7f18f4074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:55.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 1 -- 172.21.15.37:0/52864408 shutdown_connections 2024-03-25T18:17:55.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 1 -- 172.21.15.37:0/52864408 wait complete. 2024-03-25T18:17:55.389 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:17:55.386+0000 7f18f9286640 1 librados: shutdown 2024-03-25T18:17:55.398 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:56.179 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:10:31.524 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:56.223 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:10:31.536 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:56.225 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:10:31.538 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:56.227 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:10:31.572 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:56.249 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:10:31.562 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:56.251 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:10:31.596 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:56.302 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:10:31.647 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:56.307 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:10:31.652 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:56.343 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:10:31.688 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:02.205 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:02.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.320+0000 7fba6b60e640 10 monclient: get_monmap_and_config 2024-03-25T18:18:02.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.320+0000 7fba6b60e640 10 monclient: build_initial_monmap 2024-03-25T18:18:02.321 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.320+0000 7fba6b60e640 10 monclient: monmap: 2024-03-25T18:18:02.321 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:02.322 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:02.322 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:18:02.321274+0000 2024-03-25T18:18:02.322 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:18:02.321274+0000 2024-03-25T18:18:02.322 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:02.322 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:02.322 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:02.322 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:18:02.322 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:18:02.322 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:02.323 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.321+0000 7fba6b60e640 1 Processor -- start 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.321+0000 7fba6b60e640 1 -- start start 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.321+0000 7fba6b60e640 10 monclient: init 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.322+0000 7fba6b60e640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.322+0000 7fba6b60e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.322+0000 7fba6b60e640 10 monclient(hunting): picked mon.noname-a con 0x7fba640705b0 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.322+0000 7fba6b60e640 10 monclient(hunting): picked mon.noname-c con 0x7fba641402d0 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.322+0000 7fba6b60e640 10 monclient(hunting): picked mon.noname-b con 0x7fba64149ae0 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.322+0000 7fba6b60e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba640718c0 con 0x7fba640705b0 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.322+0000 7fba6b60e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba640714c0 con 0x7fba641402d0 2024-03-25T18:18:02.324 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.322+0000 7fba6b60e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba64072b50 con 0x7fba64149ae0 2024-03-25T18:18:02.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.322+0000 7fba6b60e640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:02.325 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.322+0000 7fba6b60e640 10 monclient(hunting): authenticate will time out at 7437.574458s 2024-03-25T18:18:02.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.332+0000 7fba69bab640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fba64149ae0 0x7fba6414bed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:34664/0 (socket says 172.21.15.37:34664) 2024-03-25T18:18:02.333 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.332+0000 7fba69bab640 1 -- 172.21.15.37:0/103826803 learned_addr learned my addr 172.21.15.37:0/103826803 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:02.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.340+0000 7fba5bfff640 1 -- 172.21.15.37:0/103826803 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2016987089 0 0) 0x7fba64072b50 con 0x7fba64149ae0 2024-03-25T18:18:02.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.340+0000 7fba5bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:02.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.340+0000 7fba5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:02.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.340+0000 7fba5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:02.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.340+0000 7fba5bfff640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:18:02.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.340+0000 7fba5bfff640 1 -- 172.21.15.37:0/103826803 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba50003520 con 0x7fba64149ae0 2024-03-25T18:18:02.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.344+0000 7fba5bfff640 1 -- 172.21.15.37:0/103826803 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1090672420 0 0) 0x7fba640718c0 con 0x7fba640705b0 2024-03-25T18:18:02.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.344+0000 7fba5bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:02.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.344+0000 7fba5bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:02.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.344+0000 7fba5bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:02.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.344+0000 7fba5bfff640 10 monclient(hunting): my global_id is 4695 2024-03-25T18:18:02.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.344+0000 7fba5bfff640 1 -- 172.21.15.37:0/103826803 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba64072b50 con 0x7fba640705b0 2024-03-25T18:18:02.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 1 -- 172.21.15.37:0/103826803 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3022450921 0 0) 0x7fba64072b50 con 0x7fba640705b0 2024-03-25T18:18:02.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 1 -- 172.21.15.37:0/103826803 >> v1:172.21.15.37:6790/0 conn(0x7fba64149ae0 legacy=0x7fba6414bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:02.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 1 -- 172.21.15.37:0/103826803 >> v1:172.21.15.117:6789/0 conn(0x7fba641402d0 legacy=0x7fba64140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:02.353 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:02.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 1 monclient: found mon.noname-a 2024-03-25T18:18:02.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:02.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:18:02.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 1 -- 172.21.15.37:0/103826803 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba6413dd10 con 0x7fba640705b0 2024-03-25T18:18:02.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 10 monclient: _finish_auth 0 2024-03-25T18:18:02.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:02.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:02.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba5bfff640 1 -- 172.21.15.37:0/103826803 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fba60001670 con 0x7fba640705b0 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba6b60e640 5 monclient: authenticate success, global_id 4695 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.351+0000 7fba6b60e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.352+0000 7fba5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.352+0000 7fba5bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.352+0000 7fba5bfff640 10 monclient: dump: 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:02.355 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:02.356 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:02.356 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:02.356 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:02.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.352+0000 7fba6b60e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:02.356 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.355+0000 7fba5bfff640 1 -- 172.21.15.37:0/103826803 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba600017d0 con 0x7fba640705b0 2024-03-25T18:18:02.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.355+0000 7fba5bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:02.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.355+0000 7fba5bfff640 1 -- 172.21.15.37:0/103826803 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fba60001ba0 con 0x7fba640705b0 2024-03-25T18:18:02.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.355+0000 7fba5bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:02.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.355+0000 7fba5bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:02.357 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.355+0000 7fba5bfff640 10 monclient: dump: 2024-03-25T18:18:02.357 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:02.357 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:02.357 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:02.357 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:02.357 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:02.358 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:02.358 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:02.358 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:02.358 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:02.358 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:02.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.355+0000 7fba6b60e640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:02.358 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.355+0000 7fba6b60e640 4 set_mon_vals no callback set 2024-03-25T18:18:02.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.356+0000 7fba6b60e640 10 monclient: shutdown 2024-03-25T18:18:02.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.356+0000 7fba6b60e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:02.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.356+0000 7fba6b60e640 1 -- 172.21.15.37:0/103826803 >> v1:172.21.15.37:6789/0 conn(0x7fba640705b0 legacy=0x7fba6413d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:02.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.356+0000 7fba6b60e640 1 -- 172.21.15.37:0/103826803 shutdown_connections 2024-03-25T18:18:02.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.356+0000 7fba6b60e640 1 -- 172.21.15.37:0/103826803 >> 172.21.15.37:0/103826803 conn(0x7fba640b5d10 msgr2=0x7fba640b60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:02.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.356+0000 7fba6b60e640 1 -- 172.21.15.37:0/103826803 shutdown_connections 2024-03-25T18:18:02.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.356+0000 7fba6b60e640 1 -- 172.21.15.37:0/103826803 wait complete. 2024-03-25T18:18:02.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.357+0000 7fba6b60e640 10 monclient: build_initial_monmap 2024-03-25T18:18:02.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.357+0000 7fba6b60e640 10 monclient: monmap: 2024-03-25T18:18:02.361 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:02.361 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:02.361 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:18:02.361 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:18:02.362 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:02.362 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:18:02.362 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:02.362 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:18:02.362 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:18:02.362 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:02.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.357+0000 7fba6b60e640 1 librados: starting msgr at 2024-03-25T18:18:02.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.357+0000 7fba6b60e640 1 librados: starting objecter 2024-03-25T18:18:02.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.358+0000 7fba6b60e640 1 Processor -- start 2024-03-25T18:18:02.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.358+0000 7fba6b60e640 1 -- start start 2024-03-25T18:18:02.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.358+0000 7fba6b60e640 1 librados: setting wanted keys 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.358+0000 7fba6b60e640 1 librados: calling monclient init 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.358+0000 7fba6b60e640 10 monclient: init 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.358+0000 7fba6b60e640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.358+0000 7fba6b60e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.358+0000 7fba6b60e640 10 monclient(hunting): picked mon.noname-b con 0x7fba640705b0 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.358+0000 7fba6b60e640 10 monclient(hunting): picked mon.noname-a con 0x7fba641402d0 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.359+0000 7fba6b60e640 10 monclient(hunting): picked mon.noname-c con 0x7fba64149ae0 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.359+0000 7fba6b60e640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba64142250 con 0x7fba641402d0 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.359+0000 7fba6b60e640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba64143a40 con 0x7fba640705b0 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.359+0000 7fba6b60e640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba64141cb0 con 0x7fba64149ae0 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.359+0000 7fba6b60e640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:02.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.359+0000 7fba6b60e640 10 monclient(hunting): authenticate will time out at 7437.611340s 2024-03-25T18:18:02.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.368+0000 7fba693aa640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fba640705b0 0x7fba641d7300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:38526/0 (socket says 172.21.15.37:38526) 2024-03-25T18:18:02.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.368+0000 7fba693aa640 1 -- 172.21.15.37:0/2314286090 learned_addr learned my addr 172.21.15.37:0/2314286090 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:02.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.371+0000 7fba59ffb640 10 client.?.objecter ms_handle_connect 0x7fba640705b0 2024-03-25T18:18:02.372 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.371+0000 7fba59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:02.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.375+0000 7fba59ffb640 10 client.?.objecter ms_handle_connect 0x7fba641402d0 2024-03-25T18:18:02.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.375+0000 7fba59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:02.384 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.383+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592255563 0 0) 0x7fba64142250 con 0x7fba641402d0 2024-03-25T18:18:02.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.383+0000 7fba59ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:02.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.383+0000 7fba59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:02.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.383+0000 7fba59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:02.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.383+0000 7fba59ffb640 10 monclient(hunting): my global_id is 4698 2024-03-25T18:18:02.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.383+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba48004eb0 con 0x7fba641402d0 2024-03-25T18:18:02.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.385+0000 7fba59ffb640 10 client.?.objecter ms_handle_connect 0x7fba64149ae0 2024-03-25T18:18:02.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.385+0000 7fba59ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:02.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.394+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2232123555 0 0) 0x7fba64143a40 con 0x7fba640705b0 2024-03-25T18:18:02.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.394+0000 7fba59ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:02.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.394+0000 7fba59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:02.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.394+0000 7fba59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:02.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.394+0000 7fba59ffb640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:18:02.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.394+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba64142250 con 0x7fba640705b0 2024-03-25T18:18:02.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.395+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 437598416 0 0) 0x7fba64141cb0 con 0x7fba64149ae0 2024-03-25T18:18:02.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.395+0000 7fba59ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:02.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.395+0000 7fba59ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:02.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.395+0000 7fba59ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:02.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.395+0000 7fba59ffb640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:18:02.397 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.395+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba64143a40 con 0x7fba64149ae0 2024-03-25T18:18:02.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.403+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 603157112 0 0) 0x7fba48004eb0 con 0x7fba641402d0 2024-03-25T18:18:02.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.404+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba64141cb0 con 0x7fba641402d0 2024-03-25T18:18:02.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.404+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fba540018c0 con 0x7fba641402d0 2024-03-25T18:18:02.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.404+0000 7fba59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:02.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.404+0000 7fba59ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:02.405 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.404+0000 7fba59ffb640 10 monclient(hunting): dump: 2024-03-25T18:18:02.405 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:02.405 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:02.406 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:02.406 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:02.406 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:02.406 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:02.406 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:02.406 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:02.406 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:02.406 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:02.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.404+0000 7fba59ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:02.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.408+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1519178773 0 0) 0x7fba64143a40 con 0x7fba64149ae0 2024-03-25T18:18:02.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.408+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba48004eb0 con 0x7fba64149ae0 2024-03-25T18:18:02.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.408+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fba5c001c10 con 0x7fba64149ae0 2024-03-25T18:18:02.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.408+0000 7fba59ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:02.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.408+0000 7fba59ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:02.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.408+0000 7fba59ffb640 10 monclient(hunting): dump: 2024-03-25T18:18:02.409 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:02.409 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:02.409 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:02.410 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:02.410 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:02.410 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:02.410 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:02.410 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:02.410 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:02.410 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:02.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1894159530 0 0) 0x7fba48004eb0 con 0x7fba64149ae0 2024-03-25T18:18:02.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 >> v1:172.21.15.117:6789/0 conn(0x7fba640705b0 legacy=0x7fba641d7300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 >> v1:172.21.15.37:6789/0 conn(0x7fba641402d0 legacy=0x7fba641426e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba59ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba59ffb640 1 monclient: found mon.c 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba59ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba59ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba64148ee0 con 0x7fba64149ae0 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba59ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba59ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba59ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba6b60e640 5 monclient: authenticate success, global_id 4682 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba6b60e640 10 monclient: _renew_subs 2024-03-25T18:18:02.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba6b60e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:02.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba6b60e640 1 -- 172.21.15.37:0/2314286090 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba64149190 con 0x7fba64149ae0 2024-03-25T18:18:02.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba6b60e640 10 client.4682.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:02.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba6b60e640 10 monclient: _renew_subs 2024-03-25T18:18:02.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba6b60e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:02.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba6b60e640 1 -- 172.21.15.37:0/2314286090 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba64144320 con 0x7fba64149ae0 2024-03-25T18:18:02.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.426+0000 7fba6b60e640 1 librados: init done 2024-03-25T18:18:02.429 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.427+0000 7fba6b60e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:02.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.427+0000 7fba6b60e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:02.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.427+0000 7fba6b60e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:02.430 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.427+0000 7fba6b60e640 1 -- 172.21.15.37:0/2314286090 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba64142020 con 0x7fba64149ae0 2024-03-25T18:18:02.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.764+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba5c001310 con 0x7fba64149ae0 2024-03-25T18:18:02.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.764+0000 7fba59ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:02.765 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.764+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fba5c0020e0 con 0x7fba64149ae0 2024-03-25T18:18:02.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.764+0000 7fba59ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:02.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.764+0000 7fba59ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:02.766 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.764+0000 7fba59ffb640 10 monclient: dump: 2024-03-25T18:18:02.766 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:02.766 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:02.766 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:02.766 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:02.766 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:02.766 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:02.766 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:02.766 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:02.767 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:02.767 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:02.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.764+0000 7fba5b7fe640 4 set_mon_vals no callback set 2024-03-25T18:18:02.767 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.765+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fba5c01aa90 con 0x7fba64149ae0 2024-03-25T18:18:02.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.765+0000 7fba693aa640 10 monclient: get_auth_request con 0x7fba4806df70 auth_method 0 2024-03-25T18:18:02.768 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.766+0000 7fba59ffb640 10 client.4682.objecter ms_handle_connect 0x7fba4806df70 2024-03-25T18:18:02.773 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.772+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7fba5c0547b0 con 0x7fba64149ae0 2024-03-25T18:18:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.772+0000 7fba59ffb640 10 client.4682.objecter ms_dispatch 0x7fba64000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.772+0000 7fba59ffb640 3 client.4682.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.772+0000 7fba59ffb640 3 client.4682.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.772+0000 7fba59ffb640 20 client.4682.objecter dump_active .. 0 homeless 2024-03-25T18:18:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.772+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba5c0500f0 con 0x7fba64149ae0 2024-03-25T18:18:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.772+0000 7fba59ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:02.774 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.772+0000 7fba59ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:02.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.993+0000 7fba6b60e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:02.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.993+0000 7fba6b60e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:02.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.993+0000 7fba6b60e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:02.995 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:02.993+0000 7fba6b60e640 1 -- 172.21.15.37:0/2314286090 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba64142400 con 0x7fba64149ae0 2024-03-25T18:18:03.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.030+0000 7fba59ffb640 1 -- 172.21.15.37:0/2314286090 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7fba64142400 con 0x7fba64149ae0 2024-03-25T18:18:03.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.030+0000 7fba59ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:03.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.030+0000 7fba59ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:03.032 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:18:03.032 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:03.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.032+0000 7fba6b60e640 10 librados: watch_flush enter 2024-03-25T18:18:03.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.032+0000 7fba6b60e640 10 librados: watch_flush exit 2024-03-25T18:18:03.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.032+0000 7fba6b60e640 20 client.4682.objecter shutdown clearing up homeless session... 2024-03-25T18:18:03.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.032+0000 7fba6b60e640 10 client.4682.objecter successfully canceled tick 2024-03-25T18:18:03.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.033+0000 7fba6b60e640 1 -- 172.21.15.37:0/2314286090 >> v1:172.21.15.117:6800/3703993921 conn(0x7fba4806df70 legacy=0x7fba48070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:03.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.033+0000 7fba6b60e640 10 monclient: shutdown 2024-03-25T18:18:03.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.033+0000 7fba6b60e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:03.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.033+0000 7fba6b60e640 1 -- 172.21.15.37:0/2314286090 >> v1:172.21.15.37:6790/0 conn(0x7fba64149ae0 legacy=0x7fba641e1e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:03.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.033+0000 7fba6b60e640 1 -- 172.21.15.37:0/2314286090 shutdown_connections 2024-03-25T18:18:03.035 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.033+0000 7fba6b60e640 1 -- 172.21.15.37:0/2314286090 >> 172.21.15.37:0/2314286090 conn(0x7fba640b5d10 msgr2=0x7fba6413c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:03.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.033+0000 7fba6b60e640 1 -- 172.21.15.37:0/2314286090 shutdown_connections 2024-03-25T18:18:03.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.033+0000 7fba6b60e640 1 -- 172.21.15.37:0/2314286090 wait complete. 2024-03-25T18:18:03.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:03.033+0000 7fba6b60e640 1 librados: shutdown 2024-03-25T18:18:03.046 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:09.854 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:09.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.967+0000 7f0a26893640 10 monclient: get_monmap_and_config 2024-03-25T18:18:09.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.967+0000 7f0a26893640 10 monclient: build_initial_monmap 2024-03-25T18:18:09.968 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.967+0000 7f0a26893640 10 monclient: monmap: 2024-03-25T18:18:09.968 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:09.968 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:09.968 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:18:09.968133+0000 2024-03-25T18:18:09.968 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:18:09.968133+0000 2024-03-25T18:18:09.969 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:09.969 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:09.969 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:09.969 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:18:09.969 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:18:09.969 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:09.970 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.968+0000 7f0a26893640 1 Processor -- start 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.968+0000 7f0a26893640 1 -- start start 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.968+0000 7f0a26893640 10 monclient: init 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.969+0000 7f0a26893640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.969+0000 7f0a26893640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.969+0000 7f0a26893640 10 monclient(hunting): picked mon.noname-a con 0x7f0a200a98f0 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.969+0000 7f0a26893640 10 monclient(hunting): picked mon.noname-b con 0x7f0a2013f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.969+0000 7f0a26893640 10 monclient(hunting): picked mon.noname-c con 0x7f0a201430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.969+0000 7f0a26893640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a20142520 con 0x7f0a200a98f0 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.969+0000 7f0a26893640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a200b28a0 con 0x7f0a201430c0 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.969+0000 7f0a26893640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a200b2a30 con 0x7f0a2013f710 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.969+0000 7f0a26893640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:09.971 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.969+0000 7f0a26893640 10 monclient(hunting): authenticate will time out at 7445.221693s 2024-03-25T18:18:09.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.979+0000 7f0a1ffff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0a200a98f0 0x7f0a200a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:45648/0 (socket says 172.21.15.37:45648) 2024-03-25T18:18:09.980 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:09.979+0000 7f0a1ffff640 1 -- 172.21.15.37:0/3489600936 learned_addr learned my addr 172.21.15.37:0/3489600936 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:10.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.002+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1763923951 0 0) 0x7f0a200b2a30 con 0x7f0a2013f710 2024-03-25T18:18:10.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.002+0000 7f0a1effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:10.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.002+0000 7f0a1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:10.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.002+0000 7f0a1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:10.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.002+0000 7f0a1effd640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:18:10.004 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.002+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a08003520 con 0x7f0a2013f710 2024-03-25T18:18:10.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.005+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796159033 0 0) 0x7f0a200b28a0 con 0x7f0a201430c0 2024-03-25T18:18:10.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.005+0000 7f0a1effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:10.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.005+0000 7f0a1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:10.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.005+0000 7f0a1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:10.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.005+0000 7f0a1effd640 10 monclient(hunting): my global_id is 4687 2024-03-25T18:18:10.007 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.005+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a200b2a30 con 0x7f0a201430c0 2024-03-25T18:18:10.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.010+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019371957 0 0) 0x7f0a20142520 con 0x7f0a200a98f0 2024-03-25T18:18:10.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.010+0000 7f0a1effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:10.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.010+0000 7f0a1effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:10.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.010+0000 7f0a1effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:10.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.010+0000 7f0a1effd640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:18:10.011 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.010+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a200b28a0 con 0x7f0a200a98f0 2024-03-25T18:18:10.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.034+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 29639637 0 0) 0x7f0a08003520 con 0x7f0a2013f710 2024-03-25T18:18:10.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.034+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 >> v1:172.21.15.117:6789/0 conn(0x7f0a201430c0 legacy=0x7f0a2014dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:10.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.034+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 >> v1:172.21.15.37:6789/0 conn(0x7f0a200a98f0 legacy=0x7f0a200a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:10.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.034+0000 7f0a1effd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:10.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.034+0000 7f0a1effd640 1 monclient: found mon.noname-b 2024-03-25T18:18:10.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.034+0000 7f0a1effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:10.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.034+0000 7f0a1effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:18:10.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.034+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a200b44c0 con 0x7f0a2013f710 2024-03-25T18:18:10.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.035+0000 7f0a1effd640 10 monclient: _finish_auth 0 2024-03-25T18:18:10.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.035+0000 7f0a1effd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:10.036 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.035+0000 7f0a1effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.035+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0a14001690 con 0x7f0a2013f710 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.035+0000 7f0a1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.035+0000 7f0a1effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.035+0000 7f0a1effd640 10 monclient: dump: 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.035+0000 7f0a26893640 5 monclient: authenticate success, global_id 4685 2024-03-25T18:18:10.037 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.035+0000 7f0a26893640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:10.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a140017f0 con 0x7f0a2013f710 2024-03-25T18:18:10.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a1effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:10.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a1effd640 1 -- 172.21.15.37:0/3489600936 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0a14001c00 con 0x7f0a2013f710 2024-03-25T18:18:10.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a26893640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:10.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a26893640 4 set_mon_vals no callback set 2024-03-25T18:18:10.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a1effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a1effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a1effd640 10 monclient: dump: 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:10.059 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:10.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a26893640 10 monclient: shutdown 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a26893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a26893640 1 -- 172.21.15.37:0/3489600936 >> v1:172.21.15.37:6790/0 conn(0x7f0a2013f710 legacy=0x7f0a201429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a26893640 1 -- 172.21.15.37:0/3489600936 shutdown_connections 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.057+0000 7f0a26893640 1 -- 172.21.15.37:0/3489600936 >> 172.21.15.37:0/3489600936 conn(0x7f0a200b7ea0 msgr2=0x7f0a200b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.058+0000 7f0a26893640 1 -- 172.21.15.37:0/3489600936 shutdown_connections 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.058+0000 7f0a26893640 1 -- 172.21.15.37:0/3489600936 wait complete. 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.058+0000 7f0a26893640 10 monclient: build_initial_monmap 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.059+0000 7f0a26893640 10 monclient: monmap: 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:18:10.062 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.059+0000 7f0a26893640 1 librados: starting msgr at 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.059+0000 7f0a26893640 1 librados: starting objecter 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.059+0000 7f0a26893640 1 Processor -- start 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.059+0000 7f0a26893640 1 -- start start 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.059+0000 7f0a26893640 1 librados: setting wanted keys 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.059+0000 7f0a26893640 1 librados: calling monclient init 2024-03-25T18:18:10.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.059+0000 7f0a26893640 10 monclient: init 2024-03-25T18:18:10.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.060+0000 7f0a26893640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:10.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.060+0000 7f0a26893640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:10.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.060+0000 7f0a26893640 10 monclient(hunting): picked mon.noname-c con 0x7f0a200a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:10.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.060+0000 7f0a26893640 10 monclient(hunting): picked mon.noname-b con 0x7f0a2013f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:10.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.060+0000 7f0a26893640 10 monclient(hunting): picked mon.noname-a con 0x7f0a201430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:10.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.060+0000 7f0a26893640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a200ab8b0 con 0x7f0a201430c0 2024-03-25T18:18:10.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.060+0000 7f0a26893640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a200abb90 con 0x7f0a2013f710 2024-03-25T18:18:10.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.060+0000 7f0a26893640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0a20137870 con 0x7f0a200a98f0 2024-03-25T18:18:10.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.060+0000 7f0a26893640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:10.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.060+0000 7f0a26893640 10 monclient(hunting): authenticate will time out at 7445.312590s 2024-03-25T18:18:10.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.081+0000 7f0a24e30640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f0a201430c0 0x7f0a201e29a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:45720/0 (socket says 172.21.15.37:45720) 2024-03-25T18:18:10.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.081+0000 7f0a24e30640 1 -- 172.21.15.37:0/669056184 learned_addr learned my addr 172.21.15.37:0/669056184 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:10.082 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.081+0000 7f0a1cff9640 10 client.?.objecter ms_handle_connect 0x7f0a201430c0 2024-03-25T18:18:10.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.081+0000 7f0a1cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:10.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.087+0000 7f0a1cff9640 10 client.?.objecter ms_handle_connect 0x7f0a2013f710 2024-03-25T18:18:10.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.087+0000 7f0a1cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:10.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.097+0000 7f0a1cff9640 10 client.?.objecter ms_handle_connect 0x7f0a200a98f0 2024-03-25T18:18:10.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.097+0000 7f0a1cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:10.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.098+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2866935597 0 0) 0x7f0a200abb90 con 0x7f0a2013f710 2024-03-25T18:18:10.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.098+0000 7f0a1cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:10.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.098+0000 7f0a1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:10.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.098+0000 7f0a1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:10.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.098+0000 7f0a1cff9640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:18:10.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.098+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0c004eb0 con 0x7f0a2013f710 2024-03-25T18:18:10.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.100+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 231340896 0 0) 0x7f0a200ab8b0 con 0x7f0a201430c0 2024-03-25T18:18:10.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.100+0000 7f0a1cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:10.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.100+0000 7f0a1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:10.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.100+0000 7f0a1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:10.101 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.100+0000 7f0a1cff9640 10 monclient(hunting): my global_id is 4704 2024-03-25T18:18:10.102 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.100+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a200abb90 con 0x7f0a201430c0 2024-03-25T18:18:10.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.113+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 132357613 0 0) 0x7f0a200abb90 con 0x7f0a201430c0 2024-03-25T18:18:10.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.113+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a200ab8b0 con 0x7f0a201430c0 2024-03-25T18:18:10.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.113+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0a18001c10 con 0x7f0a201430c0 2024-03-25T18:18:10.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.113+0000 7f0a1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:10.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.113+0000 7f0a1cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:10.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.113+0000 7f0a1cff9640 10 monclient(hunting): dump: 2024-03-25T18:18:10.114 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:10.114 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:10.115 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:10.115 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:10.115 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:10.115 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:10.115 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:10.115 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:10.115 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:10.115 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:10.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.113+0000 7f0a1cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:10.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.119+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4041530639 0 0) 0x7f0a0c004eb0 con 0x7f0a2013f710 2024-03-25T18:18:10.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.119+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0a200abb90 con 0x7f0a2013f710 2024-03-25T18:18:10.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.119+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0a14001b00 con 0x7f0a2013f710 2024-03-25T18:18:10.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.119+0000 7f0a1cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:10.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.119+0000 7f0a1cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:10.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.119+0000 7f0a1cff9640 10 monclient(hunting): dump: 2024-03-25T18:18:10.121 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:10.121 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:10.121 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:10.121 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:10.121 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:10.121 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:10.121 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:10.121 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:10.121 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:10.121 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:10.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.120+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2676888194 0 0) 0x7f0a20137870 con 0x7f0a200a98f0 2024-03-25T18:18:10.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.120+0000 7f0a1cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:10.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.120+0000 7f0a1cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:10.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.120+0000 7f0a1cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:10.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.120+0000 7f0a1cff9640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:18:10.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.120+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0a0c004eb0 con 0x7f0a200a98f0 2024-03-25T18:18:10.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4109207801 0 0) 0x7f0a200ab8b0 con 0x7f0a201430c0 2024-03-25T18:18:10.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 >> v1:172.21.15.37:6790/0 conn(0x7f0a200a98f0 legacy=0x7f0a201350d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:10.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 >> v1:172.21.15.117:6789/0 conn(0x7f0a2013f710 legacy=0x7f0a201df270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:10.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a1cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:10.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a1cff9640 1 monclient: found mon.a 2024-03-25T18:18:10.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a1cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:10.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a1cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:18:10.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a201389e0 con 0x7f0a201430c0 2024-03-25T18:18:10.129 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a1cff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:10.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a1cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:10.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a1cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:10.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a26893640 5 monclient: authenticate success, global_id 4704 2024-03-25T18:18:10.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a26893640 10 monclient: _renew_subs 2024-03-25T18:18:10.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a26893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:18:10.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a26893640 1 -- 172.21.15.37:0/669056184 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a20136d90 con 0x7f0a201430c0 2024-03-25T18:18:10.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a26893640 10 client.4704.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:10.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a26893640 10 monclient: _renew_subs 2024-03-25T18:18:10.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a26893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:18:10.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a26893640 1 -- 172.21.15.37:0/669056184 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a201384c0 con 0x7f0a201430c0 2024-03-25T18:18:10.130 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.127+0000 7f0a26893640 1 librados: init done 2024-03-25T18:18:10.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.128+0000 7f0a26893640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:10.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.128+0000 7f0a26893640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:10.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.128+0000 7f0a26893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:18:10.131 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.128+0000 7f0a26893640 1 -- 172.21.15.37:0/669056184 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a200627b0 con 0x7f0a201430c0 2024-03-25T18:18:10.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.461+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0a18003970 con 0x7f0a201430c0 2024-03-25T18:18:10.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.461+0000 7f0a1cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:10.462 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.461+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0a18003dc0 con 0x7f0a201430c0 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.461+0000 7f0a1cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.461+0000 7f0a1cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.461+0000 7f0a1cff9640 10 monclient: dump: 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:10.463 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.461+0000 7f0a1e7fc640 4 set_mon_vals no callback set 2024-03-25T18:18:10.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.462+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f0a18003fe0 con 0x7f0a201430c0 2024-03-25T18:18:10.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.462+0000 7f0a1ffff640 10 monclient: get_auth_request con 0x7f0a0c06df00 auth_method 0 2024-03-25T18:18:10.464 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.463+0000 7f0a1cff9640 10 client.4704.objecter ms_handle_connect 0x7f0a0c06df00 2024-03-25T18:18:10.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.467+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7f0a18054a00 con 0x7f0a201430c0 2024-03-25T18:18:10.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.468+0000 7f0a1cff9640 10 client.4704.objecter ms_dispatch 0x7f0a20000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:10.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.468+0000 7f0a1cff9640 3 client.4704.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:10.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.468+0000 7f0a1cff9640 3 client.4704.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:10.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.468+0000 7f0a1cff9640 20 client.4704.objecter dump_active .. 0 homeless 2024-03-25T18:18:10.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.468+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0a18050510 con 0x7f0a201430c0 2024-03-25T18:18:10.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.468+0000 7f0a1cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:10.469 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.468+0000 7f0a1cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:10.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.697+0000 7f0a26893640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:10.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.697+0000 7f0a26893640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:10.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.697+0000 7f0a26893640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:18:10.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.697+0000 7f0a26893640 1 -- 172.21.15.37:0/669056184 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0a20138b70 con 0x7f0a201430c0 2024-03-25T18:18:10.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.724+0000 7f0a1cff9640 1 -- 172.21.15.37:0/669056184 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7f0a18052440 con 0x7f0a201430c0 2024-03-25T18:18:10.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.724+0000 7f0a1cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:10.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.724+0000 7f0a1cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:10.726 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:18:10.726 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:10.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.727+0000 7f0a26893640 10 librados: watch_flush enter 2024-03-25T18:18:10.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.727+0000 7f0a26893640 10 librados: watch_flush exit 2024-03-25T18:18:10.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.727+0000 7f0a26893640 20 client.4704.objecter shutdown clearing up homeless session... 2024-03-25T18:18:10.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.727+0000 7f0a26893640 10 client.4704.objecter successfully canceled tick 2024-03-25T18:18:10.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.727+0000 7f0a26893640 1 -- 172.21.15.37:0/669056184 >> v1:172.21.15.117:6800/3703993921 conn(0x7f0a0c06df00 legacy=0x7f0a0c0703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:10.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.727+0000 7f0a26893640 10 monclient: shutdown 2024-03-25T18:18:10.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.727+0000 7f0a26893640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:10.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.727+0000 7f0a26893640 1 -- 172.21.15.37:0/669056184 >> v1:172.21.15.37:6789/0 conn(0x7f0a201430c0 legacy=0x7f0a201e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:10.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.727+0000 7f0a26893640 1 -- 172.21.15.37:0/669056184 shutdown_connections 2024-03-25T18:18:10.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.727+0000 7f0a26893640 1 -- 172.21.15.37:0/669056184 >> 172.21.15.37:0/669056184 conn(0x7f0a200b7ea0 msgr2=0x7f0a20074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:10.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.728+0000 7f0a26893640 1 -- 172.21.15.37:0/669056184 shutdown_connections 2024-03-25T18:18:10.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.728+0000 7f0a26893640 1 -- 172.21.15.37:0/669056184 wait complete. 2024-03-25T18:18:10.730 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:10.728+0000 7f0a26893640 1 librados: shutdown 2024-03-25T18:18:10.739 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:16.266 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:10:51.611 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:16.269 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:10:51.614 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:16.306 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:10:51.651 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:16.322 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:10:51.635 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:16.326 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:10:51.639 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:16.352 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:10:51.665 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:16.352 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:10:51.697 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:16.363 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:10:51.708 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:16.393 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:10:51.739 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:17.546 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:17.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.656+0000 7fe044ae2640 10 monclient: get_monmap_and_config 2024-03-25T18:18:17.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.656+0000 7fe044ae2640 10 monclient: build_initial_monmap 2024-03-25T18:18:17.658 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.656+0000 7fe044ae2640 10 monclient: monmap: 2024-03-25T18:18:17.658 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:17.659 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:17.659 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:18:17.658348+0000 2024-03-25T18:18:17.659 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:18:17.658348+0000 2024-03-25T18:18:17.659 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:17.659 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:17.659 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:17.659 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:18:17.659 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:18:17.659 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:17.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.657+0000 7fe044ae2640 1 Processor -- start 2024-03-25T18:18:17.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.657+0000 7fe044ae2640 1 -- start start 2024-03-25T18:18:17.660 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.657+0000 7fe044ae2640 10 monclient: init 2024-03-25T18:18:17.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.658+0000 7fe044ae2640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:17.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.658+0000 7fe044ae2640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:17.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.658+0000 7fe044ae2640 10 monclient(hunting): picked mon.noname-b con 0x7fe0400b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:17.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.658+0000 7fe044ae2640 10 monclient(hunting): picked mon.noname-a con 0x7fe0400b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:17.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.658+0000 7fe044ae2640 10 monclient(hunting): picked mon.noname-c con 0x7fe04014be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:17.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.658+0000 7fe044ae2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0400af940 con 0x7fe0400b1730 2024-03-25T18:18:17.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.658+0000 7fe044ae2640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0400b0a70 con 0x7fe04014be90 2024-03-25T18:18:17.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.658+0000 7fe044ae2640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0400b0d50 con 0x7fe0400b4d60 2024-03-25T18:18:17.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.658+0000 7fe044ae2640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:17.661 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.658+0000 7fe044ae2640 10 monclient(hunting): authenticate will time out at 7452.911804s 2024-03-25T18:18:17.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.664+0000 7fe03e59c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fe0400b4d60 0x7fe04013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:51544/0 (socket says 172.21.15.37:51544) 2024-03-25T18:18:17.666 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.664+0000 7fe03e59c640 1 -- 172.21.15.37:0/2072647154 learned_addr learned my addr 172.21.15.37:0/2072647154 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:17.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.677+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120204798 0 0) 0x7fe0400b0d50 con 0x7fe0400b4d60 2024-03-25T18:18:17.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.678+0000 7fe03dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.678+0000 7fe03dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.678+0000 7fe03dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.678+0000 7fe03dd9b640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:18:17.680 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.678+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe02c003520 con 0x7fe0400b4d60 2024-03-25T18:18:17.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.694+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559541836 0 0) 0x7fe0400b0a70 con 0x7fe04014be90 2024-03-25T18:18:17.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.694+0000 7fe03dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.694+0000 7fe03dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.694+0000 7fe03dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.694+0000 7fe03dd9b640 10 monclient(hunting): my global_id is 4693 2024-03-25T18:18:17.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.694+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0400b0d50 con 0x7fe04014be90 2024-03-25T18:18:17.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.703+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413185543 0 0) 0x7fe0400af940 con 0x7fe0400b1730 2024-03-25T18:18:17.705 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.703+0000 7fe03dd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.703+0000 7fe03dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.703+0000 7fe03dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.703+0000 7fe03dd9b640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:18:17.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.703+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0400b0a70 con 0x7fe0400b1730 2024-03-25T18:18:17.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3123544869 0 0) 0x7fe0400b0d50 con 0x7fe04014be90 2024-03-25T18:18:17.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 >> v1:172.21.15.37:6790/0 conn(0x7fe0400b4d60 legacy=0x7fe04013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 >> v1:172.21.15.37:6789/0 conn(0x7fe0400b1730 legacy=0x7fe0400b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:18:17.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 1 monclient: found mon.noname-c 2024-03-25T18:18:17.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:17.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:18:17.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0400b1030 con 0x7fe04014be90 2024-03-25T18:18:17.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 10 monclient: _finish_auth 0 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe0380019f0 con 0x7fe04014be90 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe044ae2640 5 monclient: authenticate success, global_id 4693 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe044ae2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe03dd9b640 10 monclient: dump: 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:17.710 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:17.711 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:17.711 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:17.711 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:17.711 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:17.711 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:17.711 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:17.711 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.706+0000 7fe044ae2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:17.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.721+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe038001b50 con 0x7fe04014be90 2024-03-25T18:18:17.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.721+0000 7fe03dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:17.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.721+0000 7fe03dd9b640 1 -- 172.21.15.37:0/2072647154 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe038001f60 con 0x7fe04014be90 2024-03-25T18:18:17.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.721+0000 7fe03dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:17.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.721+0000 7fe03dd9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:17.723 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.721+0000 7fe03dd9b640 10 monclient: dump: 2024-03-25T18:18:17.723 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:17.723 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:17.723 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:17.724 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:17.724 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:17.724 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:17.724 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:17.724 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:17.724 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:17.724 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:17.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.721+0000 7fe044ae2640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:17.724 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.721+0000 7fe044ae2640 4 set_mon_vals no callback set 2024-03-25T18:18:17.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.722+0000 7fe044ae2640 10 monclient: shutdown 2024-03-25T18:18:17.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.722+0000 7fe044ae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:17.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.722+0000 7fe044ae2640 1 -- 172.21.15.37:0/2072647154 >> v1:172.21.15.117:6789/0 conn(0x7fe04014be90 legacy=0x7fe04014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.722+0000 7fe044ae2640 1 -- 172.21.15.37:0/2072647154 shutdown_connections 2024-03-25T18:18:17.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.722+0000 7fe044ae2640 1 -- 172.21.15.37:0/2072647154 >> 172.21.15.37:0/2072647154 conn(0x7fe0400617a0 msgr2=0x7fe040061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:17.725 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.722+0000 7fe044ae2640 1 -- 172.21.15.37:0/2072647154 shutdown_connections 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.722+0000 7fe044ae2640 1 -- 172.21.15.37:0/2072647154 wait complete. 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.723+0000 7fe044ae2640 10 monclient: build_initial_monmap 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.723+0000 7fe044ae2640 10 monclient: monmap: 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:18:17.726 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 1 librados: starting msgr at 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 1 librados: starting objecter 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 1 Processor -- start 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 1 -- start start 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 1 librados: setting wanted keys 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 1 librados: calling monclient init 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 10 monclient: init 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 10 monclient(hunting): picked mon.noname-a con 0x7fe0400b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 10 monclient(hunting): picked mon.noname-b con 0x7fe0400b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 10 monclient(hunting): picked mon.noname-c con 0x7fe04014be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0401406f0 con 0x7fe0400b1730 2024-03-25T18:18:17.728 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe040140380 con 0x7fe0400b4d60 2024-03-25T18:18:17.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe040144ce0 con 0x7fe04014be90 2024-03-25T18:18:17.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:17.729 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.724+0000 7fe044ae2640 10 monclient(hunting): authenticate will time out at 7452.978178s 2024-03-25T18:18:17.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.737+0000 7fe03ed9d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fe04014be90 0x7fe0401d7d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:51580/0 (socket says 172.21.15.37:51580) 2024-03-25T18:18:17.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.737+0000 7fe03ed9d640 1 -- 172.21.15.37:0/656166628 learned_addr learned my addr 172.21.15.37:0/656166628 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:17.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.738+0000 7fe036ffd640 10 client.?.objecter ms_handle_connect 0x7fe04014be90 2024-03-25T18:18:17.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.738+0000 7fe036ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:17.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.739+0000 7fe036ffd640 10 client.?.objecter ms_handle_connect 0x7fe0400b4d60 2024-03-25T18:18:17.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.739+0000 7fe036ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:17.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.745+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1317006201 0 0) 0x7fe040140380 con 0x7fe0400b4d60 2024-03-25T18:18:17.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.745+0000 7fe036ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.745+0000 7fe036ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.745+0000 7fe036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.745+0000 7fe036ffd640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:18:17.748 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.745+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe020004eb0 con 0x7fe0400b4d60 2024-03-25T18:18:17.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.751+0000 7fe036ffd640 10 client.?.objecter ms_handle_connect 0x7fe0400b1730 2024-03-25T18:18:17.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.751+0000 7fe036ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:17.754 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.752+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3597473058 0 0) 0x7fe040144ce0 con 0x7fe04014be90 2024-03-25T18:18:17.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.752+0000 7fe036ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.752+0000 7fe036ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.752+0000 7fe036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.752+0000 7fe036ffd640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:18:17.755 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.752+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe040140380 con 0x7fe04014be90 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.756+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2911437091 0 0) 0x7fe020004eb0 con 0x7fe0400b4d60 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.756+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe040144ce0 con 0x7fe0400b4d60 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.756+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe0280018f0 con 0x7fe0400b4d60 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.756+0000 7fe036ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.756+0000 7fe036ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.756+0000 7fe036ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:17.759 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:17.760 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:17.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.756+0000 7fe036ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:17.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.757+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084967383 0 0) 0x7fe0401406f0 con 0x7fe0400b1730 2024-03-25T18:18:17.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.757+0000 7fe036ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:17.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.757+0000 7fe036ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:17.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.757+0000 7fe036ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:17.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.757+0000 7fe036ffd640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:18:17.760 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.757+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe020004eb0 con 0x7fe0400b1730 2024-03-25T18:18:17.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.773+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2210062746 0 0) 0x7fe040144ce0 con 0x7fe0400b4d60 2024-03-25T18:18:17.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 >> v1:172.21.15.37:6790/0 conn(0x7fe04014be90 legacy=0x7fe0401d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 >> v1:172.21.15.37:6789/0 conn(0x7fe0400b1730 legacy=0x7fe040131e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:17.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe036ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:17.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe036ffd640 1 monclient: found mon.b 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe036ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe036ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe040132520 con 0x7fe0400b4d60 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe036ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe036ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe036ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe044ae2640 5 monclient: authenticate success, global_id 4696 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe044ae2640 10 monclient: _renew_subs 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe044ae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe044ae2640 1 -- 172.21.15.37:0/656166628 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe040132770 con 0x7fe0400b4d60 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe044ae2640 10 client.4696.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe044ae2640 10 monclient: _renew_subs 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe044ae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe044ae2640 1 -- 172.21.15.37:0/656166628 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe04013e130 con 0x7fe0400b4d60 2024-03-25T18:18:17.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.774+0000 7fe044ae2640 1 librados: init done 2024-03-25T18:18:17.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.775+0000 7fe044ae2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:17.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.775+0000 7fe044ae2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:17.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.775+0000 7fe044ae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:17.778 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:17.775+0000 7fe044ae2640 1 -- 172.21.15.37:0/656166628 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe040140880 con 0x7fe0400b4d60 2024-03-25T18:18:18.077 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.075+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe028003610 con 0x7fe0400b4d60 2024-03-25T18:18:18.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.075+0000 7fe036ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:18.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.075+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe0280039a0 con 0x7fe0400b4d60 2024-03-25T18:18:18.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.075+0000 7fe036ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:18.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.075+0000 7fe036ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:18.078 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.075+0000 7fe036ffd640 10 monclient: dump: 2024-03-25T18:18:18.078 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:18.078 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:18.078 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:18.078 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:18.078 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:18.079 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:18.079 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:18.079 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:18.079 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:18.079 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:18.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.075+0000 7fe03d59a640 4 set_mon_vals no callback set 2024-03-25T18:18:18.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.076+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fe028003bc0 con 0x7fe0400b4d60 2024-03-25T18:18:18.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.077+0000 7fe03e59c640 10 monclient: get_auth_request con 0x7fe02006de00 auth_method 0 2024-03-25T18:18:18.080 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.077+0000 7fe036ffd640 10 client.4696.objecter ms_handle_connect 0x7fe02006de00 2024-03-25T18:18:18.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.085+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7fe028053600 con 0x7fe0400b4d60 2024-03-25T18:18:18.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.085+0000 7fe036ffd640 10 client.4696.objecter ms_dispatch 0x7fe040000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:18.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.085+0000 7fe036ffd640 3 client.4696.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:18.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.085+0000 7fe036ffd640 3 client.4696.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:18.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.085+0000 7fe036ffd640 20 client.4696.objecter dump_active .. 0 homeless 2024-03-25T18:18:18.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.085+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe02804f110 con 0x7fe0400b4d60 2024-03-25T18:18:18.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.085+0000 7fe036ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:18.087 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.085+0000 7fe036ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:18.342 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.340+0000 7fe044ae2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:18.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.340+0000 7fe044ae2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:18.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.340+0000 7fe044ae2640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:18.343 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.340+0000 7fe044ae2640 1 -- 172.21.15.37:0/656166628 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe040140510 con 0x7fe0400b4d60 2024-03-25T18:18:18.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.367+0000 7fe036ffd640 1 -- 172.21.15.37:0/656166628 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7fe028051040 con 0x7fe0400b4d60 2024-03-25T18:18:18.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.367+0000 7fe036ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:18.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.367+0000 7fe036ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:18.370 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:18:18.370 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 10 librados: watch_flush enter 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 10 librados: watch_flush exit 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 20 client.4696.objecter shutdown clearing up homeless session... 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 10 client.4696.objecter successfully canceled tick 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 1 -- 172.21.15.37:0/656166628 >> v1:172.21.15.117:6800/3703993921 conn(0x7fe02006de00 legacy=0x7fe0200702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 10 monclient: shutdown 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 1 -- 172.21.15.37:0/656166628 >> v1:172.21.15.117:6789/0 conn(0x7fe0400b4d60 legacy=0x7fe0401d7620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 1 -- 172.21.15.37:0/656166628 shutdown_connections 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 1 -- 172.21.15.37:0/656166628 >> 172.21.15.37:0/656166628 conn(0x7fe0400617a0 msgr2=0x7fe04013f430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 1 -- 172.21.15.37:0/656166628 shutdown_connections 2024-03-25T18:18:18.373 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 1 -- 172.21.15.37:0/656166628 wait complete. 2024-03-25T18:18:18.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:18.370+0000 7fe044ae2640 1 librados: shutdown 2024-03-25T18:18:18.385 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:25.193 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:25.310 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.309+0000 7fd50e0b3640 10 monclient: get_monmap_and_config 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.309+0000 7fd50e0b3640 10 monclient: build_initial_monmap 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.309+0000 7fd50e0b3640 10 monclient: monmap: 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:18:25.310598+0000 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:18:25.310598+0000 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:18:25.311 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:25.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.310+0000 7fd50e0b3640 1 Processor -- start 2024-03-25T18:18:25.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.310+0000 7fd50e0b3640 1 -- start start 2024-03-25T18:18:25.312 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.310+0000 7fd50e0b3640 10 monclient: init 2024-03-25T18:18:25.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.311+0000 7fd50e0b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:25.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.311+0000 7fd50e0b3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:25.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.311+0000 7fd50e0b3640 10 monclient(hunting): picked mon.noname-c con 0x7fd5080a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:25.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.311+0000 7fd50e0b3640 10 monclient(hunting): picked mon.noname-b con 0x7fd50813f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:25.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.311+0000 7fd50e0b3640 10 monclient(hunting): picked mon.noname-a con 0x7fd5081430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:25.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.311+0000 7fd50e0b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd508142520 con 0x7fd5081430c0 2024-03-25T18:18:25.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.311+0000 7fd50e0b3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5080b28a0 con 0x7fd5080a98f0 2024-03-25T18:18:25.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.311+0000 7fd50e0b3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5080b2a30 con 0x7fd50813f710 2024-03-25T18:18:25.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.311+0000 7fd50e0b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:25.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.311+0000 7fd50e0b3640 10 monclient(hunting): authenticate will time out at 7460.563437s 2024-03-25T18:18:25.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.325+0000 7fd506ffd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fd50813f710 0x7fd5081429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:56148/0 (socket says 172.21.15.37:56148) 2024-03-25T18:18:25.326 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.325+0000 7fd506ffd640 1 -- 172.21.15.37:0/1250883881 learned_addr learned my addr 172.21.15.37:0/1250883881 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:25.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.339+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1566254389 0 0) 0x7fd5080b28a0 con 0x7fd5080a98f0 2024-03-25T18:18:25.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.339+0000 7fd5067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:25.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.339+0000 7fd5067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:25.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.339+0000 7fd5067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:25.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.339+0000 7fd5067fc640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:18:25.340 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.339+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f0003520 con 0x7fd5080a98f0 2024-03-25T18:18:25.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.340+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1224208229 0 0) 0x7fd5080b2a30 con 0x7fd50813f710 2024-03-25T18:18:25.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.340+0000 7fd5067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:25.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.340+0000 7fd5067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:25.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.340+0000 7fd5067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:25.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.340+0000 7fd5067fc640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:18:25.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.340+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5080b28a0 con 0x7fd50813f710 2024-03-25T18:18:25.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.344+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574031812 0 0) 0x7fd508142520 con 0x7fd5081430c0 2024-03-25T18:18:25.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.344+0000 7fd5067fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:25.345 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.344+0000 7fd5067fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:25.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.344+0000 7fd5067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:25.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.344+0000 7fd5067fc640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:18:25.346 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.344+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5080b2a30 con 0x7fd5081430c0 2024-03-25T18:18:25.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3951075296 0 0) 0x7fd5080b2a30 con 0x7fd5081430c0 2024-03-25T18:18:25.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 >> v1:172.21.15.37:6790/0 conn(0x7fd50813f710 legacy=0x7fd5081429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:25.359 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 >> v1:172.21.15.117:6789/0 conn(0x7fd5080a98f0 legacy=0x7fd5080a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 1 monclient: found mon.noname-a 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5080b44c0 con 0x7fd5081430c0 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd500001930 con 0x7fd5081430c0 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd50e0b3640 5 monclient: authenticate success, global_id 4713 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd50e0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd5067fc640 10 monclient: dump: 2024-03-25T18:18:25.360 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:25.361 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:25.361 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:25.361 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:25.361 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:25.361 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:25.361 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:25.361 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:25.361 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:25.361 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:25.361 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.358+0000 7fd50e0b3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.364+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd500001a90 con 0x7fd5081430c0 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.364+0000 7fd5067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.364+0000 7fd5067fc640 1 -- 172.21.15.37:0/1250883881 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd500001ee0 con 0x7fd5081430c0 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.364+0000 7fd5067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.364+0000 7fd5067fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.364+0000 7fd5067fc640 10 monclient: dump: 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:25.366 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:25.367 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:25.367 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:25.367 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:25.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.364+0000 7fd50e0b3640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:25.367 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.364+0000 7fd50e0b3640 4 set_mon_vals no callback set 2024-03-25T18:18:25.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.365+0000 7fd50e0b3640 10 monclient: shutdown 2024-03-25T18:18:25.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.365+0000 7fd50e0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:25.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.365+0000 7fd50e0b3640 1 -- 172.21.15.37:0/1250883881 >> v1:172.21.15.37:6789/0 conn(0x7fd5081430c0 legacy=0x7fd50814dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:25.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.365+0000 7fd50e0b3640 1 -- 172.21.15.37:0/1250883881 shutdown_connections 2024-03-25T18:18:25.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.365+0000 7fd50e0b3640 1 -- 172.21.15.37:0/1250883881 >> 172.21.15.37:0/1250883881 conn(0x7fd5080b7ea0 msgr2=0x7fd5080b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:25.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.365+0000 7fd50e0b3640 1 -- 172.21.15.37:0/1250883881 shutdown_connections 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.365+0000 7fd50e0b3640 1 -- 172.21.15.37:0/1250883881 wait complete. 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.366+0000 7fd50e0b3640 10 monclient: build_initial_monmap 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.366+0000 7fd50e0b3640 10 monclient: monmap: 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 1 librados: starting msgr at 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 1 librados: starting objecter 2024-03-25T18:18:25.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 1 Processor -- start 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 1 -- start start 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 1 librados: setting wanted keys 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 1 librados: calling monclient init 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 10 monclient: init 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 10 monclient(hunting): picked mon.noname-c con 0x7fd5080a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 10 monclient(hunting): picked mon.noname-a con 0x7fd50813f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 10 monclient(hunting): picked mon.noname-b con 0x7fd5081430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd508139890 con 0x7fd50813f710 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5081dba80 con 0x7fd5081430c0 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd50813ace0 con 0x7fd5080a98f0 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.367+0000 7fd50e0b3640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:25.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.368+0000 7fd50e0b3640 10 monclient(hunting): authenticate will time out at 7460.620289s 2024-03-25T18:18:25.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.384+0000 7fd5077fe640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fd5080a98f0 0x7fd508139d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:56182/0 (socket says 172.21.15.37:56182) 2024-03-25T18:18:25.386 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.384+0000 7fd5077fe640 1 -- 172.21.15.37:0/329750399 learned_addr learned my addr 172.21.15.37:0/329750399 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:25.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.389+0000 7fd4effff640 10 client.?.objecter ms_handle_connect 0x7fd5081430c0 2024-03-25T18:18:25.390 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.389+0000 7fd4effff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:25.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.390+0000 7fd4effff640 10 client.?.objecter ms_handle_connect 0x7fd5080a98f0 2024-03-25T18:18:25.391 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.390+0000 7fd4effff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:25.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.394+0000 7fd4effff640 10 client.?.objecter ms_handle_connect 0x7fd50813f710 2024-03-25T18:18:25.395 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.394+0000 7fd4effff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:25.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.403+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 480786445 0 0) 0x7fd5081dba80 con 0x7fd5081430c0 2024-03-25T18:18:25.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.403+0000 7fd4effff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:25.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.403+0000 7fd4effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:25.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.403+0000 7fd4effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:25.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.403+0000 7fd4effff640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:18:25.404 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.403+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f4004eb0 con 0x7fd5081430c0 2024-03-25T18:18:25.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.405+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1069136794 0 0) 0x7fd50813ace0 con 0x7fd5080a98f0 2024-03-25T18:18:25.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.405+0000 7fd4effff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:25.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.405+0000 7fd4effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:25.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.405+0000 7fd4effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:25.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.405+0000 7fd4effff640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:18:25.406 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.405+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5081dba80 con 0x7fd5080a98f0 2024-03-25T18:18:25.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.413+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4090816578 0 0) 0x7fd4f4004eb0 con 0x7fd5081430c0 2024-03-25T18:18:25.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.414+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd50813ace0 con 0x7fd5081430c0 2024-03-25T18:18:25.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.414+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd500001de0 con 0x7fd5081430c0 2024-03-25T18:18:25.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.414+0000 7fd4effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:25.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.414+0000 7fd4effff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:25.415 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.414+0000 7fd4effff640 10 monclient(hunting): dump: 2024-03-25T18:18:25.415 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:25.415 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:25.415 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:25.415 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:25.415 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:25.416 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:25.416 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:25.416 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:25.416 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:25.416 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:25.416 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.414+0000 7fd4effff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:25.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.416+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 280023371 0 0) 0x7fd508139890 con 0x7fd50813f710 2024-03-25T18:18:25.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.416+0000 7fd4effff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:25.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.416+0000 7fd4effff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:25.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.416+0000 7fd4effff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:25.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.416+0000 7fd4effff640 10 monclient(hunting): my global_id is 4716 2024-03-25T18:18:25.417 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.416+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4f4004eb0 con 0x7fd50813f710 2024-03-25T18:18:25.418 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.416+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1223899442 0 0) 0x7fd5081dba80 con 0x7fd5080a98f0 2024-03-25T18:18:25.418 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.416+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd508139890 con 0x7fd5080a98f0 2024-03-25T18:18:25.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.418+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd4f80019d0 con 0x7fd5080a98f0 2024-03-25T18:18:25.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.418+0000 7fd4effff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:25.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.418+0000 7fd4effff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:25.419 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.418+0000 7fd4effff640 10 monclient(hunting): dump: 2024-03-25T18:18:25.419 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:25.419 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:25.419 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:25.419 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:25.419 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:25.420 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:25.420 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:25.420 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:25.420 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:25.420 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:25.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 47789084 0 0) 0x7fd50813ace0 con 0x7fd5081430c0 2024-03-25T18:18:25.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 >> v1:172.21.15.37:6790/0 conn(0x7fd5080a98f0 legacy=0x7fd508139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 >> v1:172.21.15.37:6789/0 conn(0x7fd50813f710 legacy=0x7fd5081dac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd4effff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd4effff640 1 monclient: found mon.b 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd4effff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd4effff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd50813afc0 con 0x7fd5081430c0 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd4effff640 10 monclient: _finish_auth 0 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd4effff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd4effff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd50e0b3640 5 monclient: authenticate success, global_id 4702 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd50e0b3640 10 monclient: _renew_subs 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd50e0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd50e0b3640 1 -- 172.21.15.37:0/329750399 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd50813b2a0 con 0x7fd5081430c0 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd50e0b3640 10 client.4702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd50e0b3640 10 monclient: _renew_subs 2024-03-25T18:18:25.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd50e0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:25.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd50e0b3640 1 -- 172.21.15.37:0/329750399 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5080b7b50 con 0x7fd5081430c0 2024-03-25T18:18:25.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.425+0000 7fd50e0b3640 1 librados: init done 2024-03-25T18:18:25.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.426+0000 7fd50e0b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:25.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.426+0000 7fd50e0b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:25.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.426+0000 7fd50e0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:25.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.426+0000 7fd50e0b3640 1 -- 172.21.15.37:0/329750399 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd50814f320 con 0x7fd5081430c0 2024-03-25T18:18:25.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.695+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd500002020 con 0x7fd5081430c0 2024-03-25T18:18:25.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.695+0000 7fd4effff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:25.696 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.695+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fd500003dc0 con 0x7fd5081430c0 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.695+0000 7fd4effff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.695+0000 7fd4effff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.695+0000 7fd4effff640 10 monclient: dump: 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:25.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.695+0000 7fd505ffb640 4 set_mon_vals no callback set 2024-03-25T18:18:25.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.696+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fd500019d90 con 0x7fd5081430c0 2024-03-25T18:18:25.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.696+0000 7fd5077fe640 10 monclient: get_auth_request con 0x7fd4f406def0 auth_method 0 2024-03-25T18:18:25.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.697+0000 7fd4effff640 10 client.4702.objecter ms_handle_connect 0x7fd4f406def0 2024-03-25T18:18:25.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.702+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7fd5000538f0 con 0x7fd5081430c0 2024-03-25T18:18:25.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.702+0000 7fd4effff640 10 client.4702.objecter ms_dispatch 0x7fd508000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:25.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.702+0000 7fd4effff640 3 client.4702.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:25.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.702+0000 7fd4effff640 3 client.4702.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:25.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.702+0000 7fd4effff640 20 client.4702.objecter dump_active .. 0 homeless 2024-03-25T18:18:25.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.702+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd50004f400 con 0x7fd5081430c0 2024-03-25T18:18:25.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.702+0000 7fd4effff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:25.704 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.702+0000 7fd4effff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:25.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.951+0000 7fd50e0b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:25.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.951+0000 7fd50e0b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:25.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.951+0000 7fd50e0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:25.952 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:25.951+0000 7fd50e0b3640 1 -- 172.21.15.37:0/329750399 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5080627b0 con 0x7fd5081430c0 2024-03-25T18:18:26.020 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.019+0000 7fd4effff640 1 -- 172.21.15.37:0/329750399 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7fd500051330 con 0x7fd5081430c0 2024-03-25T18:18:26.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.019+0000 7fd4effff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:26.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.019+0000 7fd4effff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:26.021 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:18:26.021 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 10 librados: watch_flush enter 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 10 librados: watch_flush exit 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 20 client.4702.objecter shutdown clearing up homeless session... 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 10 client.4702.objecter successfully canceled tick 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 1 -- 172.21.15.37:0/329750399 >> v1:172.21.15.117:6800/3703993921 conn(0x7fd4f406def0 legacy=0x7fd4f40703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 10 monclient: shutdown 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 1 -- 172.21.15.37:0/329750399 >> v1:172.21.15.117:6789/0 conn(0x7fd5081430c0 legacy=0x7fd5081db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 1 -- 172.21.15.37:0/329750399 shutdown_connections 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 1 -- 172.21.15.37:0/329750399 >> 172.21.15.37:0/329750399 conn(0x7fd5080b7ea0 msgr2=0x7fd508074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 1 -- 172.21.15.37:0/329750399 shutdown_connections 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 1 -- 172.21.15.37:0/329750399 wait complete. 2024-03-25T18:18:26.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:26.022+0000 7fd50e0b3640 1 librados: shutdown 2024-03-25T18:18:26.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:32.842 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:32.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.962+0000 7f453abcd640 10 monclient: get_monmap_and_config 2024-03-25T18:18:32.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.962+0000 7f453abcd640 10 monclient: build_initial_monmap 2024-03-25T18:18:32.963 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.962+0000 7f453abcd640 10 monclient: monmap: 2024-03-25T18:18:32.963 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:32.963 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:32.963 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:18:32.963030+0000 2024-03-25T18:18:32.963 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:18:32.963030+0000 2024-03-25T18:18:32.963 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:32.964 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:32.964 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:32.964 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:18:32.964 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:18:32.964 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:32.965 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.962+0000 7f453abcd640 1 Processor -- start 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.962+0000 7f453abcd640 1 -- start start 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.962+0000 7f453abcd640 10 monclient: init 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.963+0000 7f453abcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.963+0000 7f453abcd640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.963+0000 7f453abcd640 10 monclient(hunting): picked mon.noname-b con 0x7f45340a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.963+0000 7f453abcd640 10 monclient(hunting): picked mon.noname-c con 0x7f453413f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.963+0000 7f453abcd640 10 monclient(hunting): picked mon.noname-a con 0x7f45341430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.963+0000 7f453abcd640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4534142520 con 0x7f45341430c0 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.963+0000 7f453abcd640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45340b28a0 con 0x7f453413f710 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.963+0000 7f453abcd640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45340b2a30 con 0x7f45340a98f0 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.963+0000 7f453abcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:32.966 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.963+0000 7f453abcd640 10 monclient(hunting): authenticate will time out at 7468.215926s 2024-03-25T18:18:32.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.964+0000 7f4533fff640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f453413f710 0x7f45341429b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:55910/0 (socket says 172.21.15.37:55910) 2024-03-25T18:18:32.967 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.964+0000 7f4533fff640 1 -- 172.21.15.37:0/2067187199 learned_addr learned my addr 172.21.15.37:0/2067187199 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:32.974 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.973+0000 7f45337fe640 1 -- 172.21.15.37:0/2067187199 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346554024 0 0) 0x7f45340b28a0 con 0x7f453413f710 2024-03-25T18:18:32.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.973+0000 7f45337fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:32.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.973+0000 7f45337fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:32.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.973+0000 7f45337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:32.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.973+0000 7f45337fe640 10 monclient(hunting): my global_id is 4705 2024-03-25T18:18:32.975 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.973+0000 7f45337fe640 1 -- 172.21.15.37:0/2067187199 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4524003520 con 0x7f453413f710 2024-03-25T18:18:32.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 1 -- 172.21.15.37:0/2067187199 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 102211661 0 0) 0x7f4524003520 con 0x7f453413f710 2024-03-25T18:18:32.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 1 -- 172.21.15.37:0/2067187199 >> v1:172.21.15.37:6790/0 conn(0x7f45340a98f0 legacy=0x7f45340a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:32.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 1 -- 172.21.15.37:0/2067187199 >> v1:172.21.15.37:6789/0 conn(0x7f45341430c0 legacy=0x7f453414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:32.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:32.984 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 1 monclient: found mon.noname-c 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.117:6789/0 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 1 -- 172.21.15.37:0/2067187199 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45340b44c0 con 0x7f453413f710 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 1 -- 172.21.15.37:0/2067187199 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f4520001690 con 0x7f453413f710 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f453abcd640 5 monclient: authenticate success, global_id 4705 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f453abcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f45337fe640 10 monclient: dump: 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:32.985 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:32.986 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:32.986 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:32.986 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:32.986 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:32.986 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:32.986 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:32.986 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:32.986 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:32.986 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.983+0000 7f453abcd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:32.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.998+0000 7f45337fe640 1 -- 172.21.15.37:0/2067187199 <== mon.1 v1:172.21.15.117:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4520001870 con 0x7f453413f710 2024-03-25T18:18:32.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.998+0000 7f45337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:32.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.998+0000 7f45337fe640 1 -- 172.21.15.37:0/2067187199 <== mon.1 v1:172.21.15.117:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f4520001c80 con 0x7f453413f710 2024-03-25T18:18:32.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.998+0000 7f45337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:32.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.998+0000 7f45337fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:32.999 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.998+0000 7f45337fe640 10 monclient: dump: 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.998+0000 7f453abcd640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:33.000 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.998+0000 7f453abcd640 4 set_mon_vals no callback set 2024-03-25T18:18:33.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.999+0000 7f453abcd640 10 monclient: shutdown 2024-03-25T18:18:33.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.999+0000 7f453abcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:33.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.999+0000 7f453abcd640 1 -- 172.21.15.37:0/2067187199 >> v1:172.21.15.117:6789/0 conn(0x7f453413f710 legacy=0x7f45341429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:33.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.999+0000 7f453abcd640 1 -- 172.21.15.37:0/2067187199 shutdown_connections 2024-03-25T18:18:33.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.999+0000 7f453abcd640 1 -- 172.21.15.37:0/2067187199 >> 172.21.15.37:0/2067187199 conn(0x7f45340b7ea0 msgr2=0x7f45340b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:33.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.999+0000 7f453abcd640 1 -- 172.21.15.37:0/2067187199 shutdown_connections 2024-03-25T18:18:33.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:32.999+0000 7f453abcd640 1 -- 172.21.15.37:0/2067187199 wait complete. 2024-03-25T18:18:33.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.000+0000 7f453abcd640 10 monclient: build_initial_monmap 2024-03-25T18:18:33.002 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.000+0000 7f453abcd640 10 monclient: monmap: 2024-03-25T18:18:33.002 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:33.003 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:33.003 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:18:33.003 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:18:33.003 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:33.003 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:18:33.003 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:33.003 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:18:33.003 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:18:33.003 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:33.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.001+0000 7f453abcd640 1 librados: starting msgr at 2024-03-25T18:18:33.003 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.001+0000 7f453abcd640 1 librados: starting objecter 2024-03-25T18:18:33.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.001+0000 7f453abcd640 1 Processor -- start 2024-03-25T18:18:33.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.001+0000 7f453abcd640 1 -- start start 2024-03-25T18:18:33.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.001+0000 7f453abcd640 1 librados: setting wanted keys 2024-03-25T18:18:33.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.001+0000 7f453abcd640 1 librados: calling monclient init 2024-03-25T18:18:33.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.001+0000 7f453abcd640 10 monclient: init 2024-03-25T18:18:33.005 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.002+0000 7f453abcd640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:33.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.002+0000 7f453abcd640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:33.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.002+0000 7f453abcd640 10 monclient(hunting): picked mon.noname-c con 0x7f45340a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:33.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.002+0000 7f453abcd640 10 monclient(hunting): picked mon.noname-a con 0x7f453413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:33.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.002+0000 7f453abcd640 10 monclient(hunting): picked mon.noname-b con 0x7f45341430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:33.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.002+0000 7f453abcd640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4534138b20 con 0x7f453413f710 2024-03-25T18:18:33.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.002+0000 7f453abcd640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4534137a80 con 0x7f45341430c0 2024-03-25T18:18:33.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.002+0000 7f453abcd640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4534139880 con 0x7f45340a98f0 2024-03-25T18:18:33.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.002+0000 7f453abcd640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:33.006 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.002+0000 7f453abcd640 10 monclient(hunting): authenticate will time out at 7468.255213s 2024-03-25T18:18:33.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.013+0000 7f453916a640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f45341430c0 0x7f45341e2040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:55938/0 (socket says 172.21.15.37:55938) 2024-03-25T18:18:33.014 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.013+0000 7f453916a640 1 -- 172.21.15.37:0/2204676786 learned_addr learned my addr 172.21.15.37:0/2204676786 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:33.015 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.014+0000 7f45317fa640 10 client.?.objecter ms_handle_connect 0x7f45341430c0 2024-03-25T18:18:33.016 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.015+0000 7f45317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:33.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.017+0000 7f45317fa640 10 client.?.objecter ms_handle_connect 0x7f453413f710 2024-03-25T18:18:33.018 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.017+0000 7f45317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:33.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.020+0000 7f45317fa640 10 client.?.objecter ms_handle_connect 0x7f45340a98f0 2024-03-25T18:18:33.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.020+0000 7f45317fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:33.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.027+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 963884313 0 0) 0x7f4534138b20 con 0x7f453413f710 2024-03-25T18:18:33.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.027+0000 7f45317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:33.028 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.027+0000 7f45317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:33.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.027+0000 7f45317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:33.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.027+0000 7f45317fa640 10 monclient(hunting): my global_id is 4722 2024-03-25T18:18:33.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.027+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4518004eb0 con 0x7f453413f710 2024-03-25T18:18:33.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.028+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81101779 0 0) 0x7f4534139880 con 0x7f45340a98f0 2024-03-25T18:18:33.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.028+0000 7f45317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:33.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.028+0000 7f45317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:33.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.028+0000 7f45317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:33.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.028+0000 7f45317fa640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:18:33.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.028+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534138b20 con 0x7f45340a98f0 2024-03-25T18:18:33.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.032+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1491864447 0 0) 0x7f4534137a80 con 0x7f45341430c0 2024-03-25T18:18:33.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.032+0000 7f45317fa640 10 monclient(hunting): _negotiate 2024-03-25T18:18:33.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.032+0000 7f45317fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:33.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.032+0000 7f45317fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:33.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.032+0000 7f45317fa640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:18:33.034 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.032+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4534139880 con 0x7f45341430c0 2024-03-25T18:18:33.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.038+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2371868299 0 0) 0x7f4518004eb0 con 0x7f453413f710 2024-03-25T18:18:33.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.038+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4534137a80 con 0x7f453413f710 2024-03-25T18:18:33.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.038+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f4520001b80 con 0x7f453413f710 2024-03-25T18:18:33.039 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.038+0000 7f45317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.038+0000 7f45317fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.038+0000 7f45317fa640 10 monclient(hunting): dump: 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:33.040 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.038+0000 7f45317fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.045+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4286006861 0 0) 0x7f4534139880 con 0x7f45341430c0 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.045+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4518004eb0 con 0x7f45341430c0 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.045+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f452c001bd0 con 0x7f45341430c0 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.045+0000 7f45317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.045+0000 7f45317fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.045+0000 7f45317fa640 10 monclient(hunting): dump: 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:33.047 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:33.048 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:33.048 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:33.048 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:33.048 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:33.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.049+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2918519956 0 0) 0x7f4534138b20 con 0x7f45340a98f0 2024-03-25T18:18:33.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.049+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4534139880 con 0x7f45340a98f0 2024-03-25T18:18:33.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.049+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f45280018a0 con 0x7f45340a98f0 2024-03-25T18:18:33.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.049+0000 7f45317fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:33.051 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.049+0000 7f45317fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.049+0000 7f45317fa640 10 monclient(hunting): dump: 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:33.052 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.0 v1:172.21.15.37:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2859023627 0 0) 0x7f4534137a80 con 0x7f453413f710 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 >> v1:172.21.15.37:6790/0 conn(0x7f45340a98f0 legacy=0x7f4534137370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 >> v1:172.21.15.117:6789/0 conn(0x7f45341430c0 legacy=0x7f45341e2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f45317fa640 10 monclient: _finish_hunting 0 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f45317fa640 1 monclient: found mon.a 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f45317fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f45317fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4534139a10 con 0x7f453413f710 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f45317fa640 10 monclient: _finish_auth 0 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f45317fa640 10 monclient: _check_auth_tickets 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f45317fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f453abcd640 5 monclient: authenticate success, global_id 4722 2024-03-25T18:18:33.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f453abcd640 10 monclient: _renew_subs 2024-03-25T18:18:33.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f453abcd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:18:33.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f453abcd640 1 -- 172.21.15.37:0/2204676786 --> v1:172.21.15.37:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45341dab00 con 0x7f453413f710 2024-03-25T18:18:33.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f453abcd640 10 client.4722.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:33.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f453abcd640 10 monclient: _renew_subs 2024-03-25T18:18:33.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f453abcd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:18:33.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f453abcd640 1 -- 172.21.15.37:0/2204676786 --> v1:172.21.15.37:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f453413b4f0 con 0x7f453413f710 2024-03-25T18:18:33.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.050+0000 7f453abcd640 1 librados: init done 2024-03-25T18:18:33.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.051+0000 7f453abcd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:33.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.051+0000 7f453abcd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:33.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.051+0000 7f453abcd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:18:33.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.051+0000 7f453abcd640 1 -- 172.21.15.37:0/2204676786 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4534074cc0 con 0x7f453413f710 2024-03-25T18:18:33.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.057+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.0 v1:172.21.15.37:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4520001d00 con 0x7f453413f710 2024-03-25T18:18:33.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.057+0000 7f45317fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:33.059 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.057+0000 7f4532ffd640 4 set_mon_vals no callback set 2024-03-25T18:18:33.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.368+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.0 v1:172.21.15.37:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f4520004d90 con 0x7f453413f710 2024-03-25T18:18:33.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.368+0000 7f45317fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:33.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.368+0000 7f45317fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:33.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.368+0000 7f45317fa640 10 monclient: dump: 2024-03-25T18:18:33.369 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:33.369 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:33.369 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:33.369 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:33.369 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:33.370 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:33.370 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:33.370 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:33.370 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:33.370 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:33.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.368+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.0 v1:172.21.15.37:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f452001ad60 con 0x7f453413f710 2024-03-25T18:18:33.370 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.369+0000 7f4538969640 10 monclient: get_auth_request con 0x7f451806e0b0 auth_method 0 2024-03-25T18:18:33.371 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.369+0000 7f45317fa640 10 client.4722.objecter ms_handle_connect 0x7f451806e0b0 2024-03-25T18:18:33.379 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.378+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.0 v1:172.21.15.37:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7f4520054950 con 0x7f453413f710 2024-03-25T18:18:33.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.378+0000 7f45317fa640 10 client.4722.objecter ms_dispatch 0x7f4534000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:33.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.378+0000 7f45317fa640 3 client.4722.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:33.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.378+0000 7f45317fa640 3 client.4722.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:33.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.378+0000 7f45317fa640 20 client.4722.objecter dump_active .. 0 homeless 2024-03-25T18:18:33.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.378+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.0 v1:172.21.15.37:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4520050290 con 0x7f453413f710 2024-03-25T18:18:33.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.378+0000 7f45317fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:33.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.378+0000 7f45317fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:33.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.640+0000 7f453abcd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:33.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.640+0000 7f453abcd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:33.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.640+0000 7f453abcd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.37:6789/0 2024-03-25T18:18:33.641 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.640+0000 7f453abcd640 1 -- 172.21.15.37:0/2204676786 --> v1:172.21.15.37:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f45340b3e80 con 0x7f453413f710 2024-03-25T18:18:33.714 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.713+0000 7f45317fa640 1 -- 172.21.15.37:0/2204676786 <== mon.0 v1:172.21.15.37:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7f4520052390 con 0x7f453413f710 2024-03-25T18:18:33.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.713+0000 7f45317fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:33.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.713+0000 7f45317fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:33.715 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:18:33.715 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 10 librados: watch_flush enter 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 10 librados: watch_flush exit 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 20 client.4722.objecter shutdown clearing up homeless session... 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 10 client.4722.objecter successfully canceled tick 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 1 -- 172.21.15.37:0/2204676786 >> v1:172.21.15.117:6800/3703993921 conn(0x7f451806e0b0 legacy=0x7f4518070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 10 monclient: shutdown 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 1 -- 172.21.15.37:0/2204676786 >> v1:172.21.15.37:6789/0 conn(0x7f453413f710 legacy=0x7f4534139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 1 -- 172.21.15.37:0/2204676786 shutdown_connections 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 1 -- 172.21.15.37:0/2204676786 >> 172.21.15.37:0/2204676786 conn(0x7f45340b7ea0 msgr2=0x7f4534074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 1 -- 172.21.15.37:0/2204676786 shutdown_connections 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 1 -- 172.21.15.37:0/2204676786 wait complete. 2024-03-25T18:18:33.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:33.716+0000 7f453abcd640 1 librados: shutdown 2024-03-25T18:18:33.731 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:36.290 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:11.635 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:36.349 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:11.695 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:36.376 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:11.722 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:36.403 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:11.716 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:36.410 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:11.723 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:36.432 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:11.777 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:36.436 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:11.781 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:36.445 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:11.758 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:36.468 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:11.813 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:40.537 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:40.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.643+0000 7ffae5e15640 10 monclient: get_monmap_and_config 2024-03-25T18:18:40.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.643+0000 7ffae5e15640 10 monclient: build_initial_monmap 2024-03-25T18:18:40.645 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.643+0000 7ffae5e15640 10 monclient: monmap: 2024-03-25T18:18:40.645 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:40.645 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:40.645 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:18:40.644624+0000 2024-03-25T18:18:40.645 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:18:40.644624+0000 2024-03-25T18:18:40.645 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:40.645 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:40.645 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:40.645 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:18:40.645 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:18:40.645 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:40.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.644+0000 7ffae5e15640 1 Processor -- start 2024-03-25T18:18:40.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.644+0000 7ffae5e15640 1 -- start start 2024-03-25T18:18:40.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.644+0000 7ffae5e15640 10 monclient: init 2024-03-25T18:18:40.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffae5e15640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:40.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffae5e15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:40.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffae5e15640 10 monclient(hunting): picked mon.noname-c con 0x7ffae00b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:40.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffae5e15640 10 monclient(hunting): picked mon.noname-a con 0x7ffae00b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:40.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffae5e15640 10 monclient(hunting): picked mon.noname-b con 0x7ffae014be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:40.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffae5e15640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae00af940 con 0x7ffae00b1730 2024-03-25T18:18:40.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffae5e15640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae00b0a70 con 0x7ffae00b4d60 2024-03-25T18:18:40.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffae5e15640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae00b0d50 con 0x7ffae014be90 2024-03-25T18:18:40.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffae5e15640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:40.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffae5e15640 10 monclient(hunting): authenticate will time out at 7475.897492s 2024-03-25T18:18:40.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffadf7fe640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7ffae00b4d60 0x7ffae013f660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:40724/0 (socket says 172.21.15.37:40724) 2024-03-25T18:18:40.648 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.645+0000 7ffadf7fe640 1 -- 172.21.15.37:0/234782593 learned_addr learned my addr 172.21.15.37:0/234782593 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:40.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.716+0000 7ffade7fc640 1 -- 172.21.15.37:0/234782593 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3660772814 0 0) 0x7ffae00af940 con 0x7ffae00b1730 2024-03-25T18:18:40.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.716+0000 7ffade7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:40.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.716+0000 7ffade7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:40.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.716+0000 7ffade7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:40.717 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.716+0000 7ffade7fc640 10 monclient(hunting): my global_id is 4725 2024-03-25T18:18:40.718 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.716+0000 7ffade7fc640 1 -- 172.21.15.37:0/234782593 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffac8003520 con 0x7ffae00b1730 2024-03-25T18:18:40.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.734+0000 7ffade7fc640 1 -- 172.21.15.37:0/234782593 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 269758003 0 0) 0x7ffae00b0a70 con 0x7ffae00b4d60 2024-03-25T18:18:40.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.734+0000 7ffade7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:40.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.734+0000 7ffade7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:40.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.734+0000 7ffade7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:40.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.734+0000 7ffade7fc640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:18:40.735 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.734+0000 7ffade7fc640 1 -- 172.21.15.37:0/234782593 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae00af940 con 0x7ffae00b4d60 2024-03-25T18:18:40.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 1 -- 172.21.15.37:0/234782593 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173671481 0 0) 0x7ffac8003520 con 0x7ffae00b1730 2024-03-25T18:18:40.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 1 -- 172.21.15.37:0/234782593 >> v1:172.21.15.37:6790/0 conn(0x7ffae014be90 legacy=0x7ffae014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:40.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 1 -- 172.21.15.37:0/234782593 >> v1:172.21.15.117:6789/0 conn(0x7ffae00b4d60 legacy=0x7ffae013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:40.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:40.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 1 monclient: found mon.noname-a 2024-03-25T18:18:40.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:40.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:18:40.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 1 -- 172.21.15.37:0/234782593 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae00b1030 con 0x7ffae00b1730 2024-03-25T18:18:40.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:40.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:40.737 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:40.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 1 -- 172.21.15.37:0/234782593 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7ffad4001690 con 0x7ffae00b1730 2024-03-25T18:18:40.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:40.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:40.738 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffade7fc640 10 monclient: dump: 2024-03-25T18:18:40.738 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:40.738 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:40.738 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:40.738 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:40.738 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:40.738 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:40.738 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:40.739 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:40.739 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:40.739 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:40.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffae5e15640 5 monclient: authenticate success, global_id 4725 2024-03-25T18:18:40.739 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.735+0000 7ffae5e15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:40.746 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.745+0000 7ffade7fc640 1 -- 172.21.15.37:0/234782593 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffad40017f0 con 0x7ffae00b1730 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.745+0000 7ffade7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.745+0000 7ffade7fc640 1 -- 172.21.15.37:0/234782593 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7ffad4001c00 con 0x7ffae00b1730 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.745+0000 7ffae5e15640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.745+0000 7ffae5e15640 4 set_mon_vals no callback set 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.745+0000 7ffade7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.745+0000 7ffade7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.745+0000 7ffade7fc640 10 monclient: dump: 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:40.747 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:40.748 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:40.748 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:40.748 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:40.748 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:40.748 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:40.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.746+0000 7ffae5e15640 10 monclient: shutdown 2024-03-25T18:18:40.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.746+0000 7ffae5e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:40.750 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.746+0000 7ffae5e15640 1 -- 172.21.15.37:0/234782593 >> v1:172.21.15.37:6789/0 conn(0x7ffae00b1730 legacy=0x7ffae00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.746+0000 7ffae5e15640 1 -- 172.21.15.37:0/234782593 shutdown_connections 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.746+0000 7ffae5e15640 1 -- 172.21.15.37:0/234782593 >> 172.21.15.37:0/234782593 conn(0x7ffae00617a0 msgr2=0x7ffae0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.746+0000 7ffae5e15640 1 -- 172.21.15.37:0/234782593 shutdown_connections 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.746+0000 7ffae5e15640 1 -- 172.21.15.37:0/234782593 wait complete. 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.747+0000 7ffae5e15640 10 monclient: build_initial_monmap 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.747+0000 7ffae5e15640 10 monclient: monmap: 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:18:40.751 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.747+0000 7ffae5e15640 1 librados: starting msgr at 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.747+0000 7ffae5e15640 1 librados: starting objecter 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.748+0000 7ffae5e15640 1 Processor -- start 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.748+0000 7ffae5e15640 1 -- start start 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.748+0000 7ffae5e15640 1 librados: setting wanted keys 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.748+0000 7ffae5e15640 1 librados: calling monclient init 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.748+0000 7ffae5e15640 10 monclient: init 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.748+0000 7ffae5e15640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.748+0000 7ffae5e15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.748+0000 7ffae5e15640 10 monclient(hunting): picked mon.noname-a con 0x7ffae00b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:40.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.749+0000 7ffae5e15640 10 monclient(hunting): picked mon.noname-b con 0x7ffae00b4d60 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:40.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.749+0000 7ffae5e15640 10 monclient(hunting): picked mon.noname-c con 0x7ffae014be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:40.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.749+0000 7ffae5e15640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae01451e0 con 0x7ffae00b1730 2024-03-25T18:18:40.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.749+0000 7ffae5e15640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae01411f0 con 0x7ffae00b4d60 2024-03-25T18:18:40.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.749+0000 7ffae5e15640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae0141380 con 0x7ffae014be90 2024-03-25T18:18:40.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.749+0000 7ffae5e15640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:40.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.749+0000 7ffae5e15640 10 monclient(hunting): authenticate will time out at 7476.001401s 2024-03-25T18:18:40.776 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.775+0000 7ffadeffd640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7ffae00b4d60 0x7ffae01427d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:40756/0 (socket says 172.21.15.37:40756) 2024-03-25T18:18:40.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.776+0000 7ffadeffd640 1 -- 172.21.15.37:0/2782963617 learned_addr learned my addr 172.21.15.37:0/2782963617 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:40.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.776+0000 7ffac7fff640 10 client.?.objecter ms_handle_connect 0x7ffae00b4d60 2024-03-25T18:18:40.777 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.776+0000 7ffac7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:40.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.778+0000 7ffac7fff640 10 client.?.objecter ms_handle_connect 0x7ffae014be90 2024-03-25T18:18:40.779 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.778+0000 7ffac7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:40.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.786+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2939298892 0 0) 0x7ffae01411f0 con 0x7ffae00b4d60 2024-03-25T18:18:40.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.786+0000 7ffac7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:40.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.786+0000 7ffac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:40.787 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.786+0000 7ffac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:40.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.786+0000 7ffac7fff640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:18:40.788 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.786+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffacc004eb0 con 0x7ffae00b4d60 2024-03-25T18:18:40.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.788+0000 7ffac7fff640 10 client.?.objecter ms_handle_connect 0x7ffae00b1730 2024-03-25T18:18:40.789 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.788+0000 7ffac7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:40.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.795+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3353903934 0 0) 0x7ffae0141380 con 0x7ffae014be90 2024-03-25T18:18:40.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.795+0000 7ffac7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:40.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.795+0000 7ffac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:40.796 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.795+0000 7ffac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:40.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.795+0000 7ffac7fff640 10 monclient(hunting): my global_id is 4712 2024-03-25T18:18:40.797 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.795+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae01411f0 con 0x7ffae014be90 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.800+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2622047078 0 0) 0x7ffacc004eb0 con 0x7ffae00b4d60 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.800+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffae0141380 con 0x7ffae00b4d60 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.800+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7ffad4001690 con 0x7ffae00b4d60 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.800+0000 7ffac7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.800+0000 7ffac7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.800+0000 7ffac7fff640 10 monclient(hunting): dump: 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:40.802 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:40.803 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:40.803 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:40.803 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:40.803 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:40.803 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.800+0000 7ffac7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.809+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3992659394 0 0) 0x7ffae01411f0 con 0x7ffae014be90 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.809+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffacc004eb0 con 0x7ffae014be90 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.809+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7ffad8001b00 con 0x7ffae014be90 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.809+0000 7ffac7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.809+0000 7ffac7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.809+0000 7ffac7fff640 10 monclient(hunting): dump: 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:40.811 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:40.812 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:40.812 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:40.812 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:40.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.814+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3373342882 0 0) 0x7ffae01451e0 con 0x7ffae00b1730 2024-03-25T18:18:40.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.814+0000 7ffac7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:40.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.814+0000 7ffac7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:40.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.814+0000 7ffac7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:40.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.814+0000 7ffac7fff640 10 monclient(hunting): my global_id is 4728 2024-03-25T18:18:40.815 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.814+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffae01411f0 con 0x7ffae00b1730 2024-03-25T18:18:40.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 246198887 0 0) 0x7ffae0141380 con 0x7ffae00b4d60 2024-03-25T18:18:40.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 >> v1:172.21.15.37:6790/0 conn(0x7ffae014be90 legacy=0x7ffae0142ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:40.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 >> v1:172.21.15.37:6789/0 conn(0x7ffae00b1730 legacy=0x7ffae0140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:40.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffac7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:40.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffac7fff640 1 monclient: found mon.b 2024-03-25T18:18:40.819 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffac7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffac7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae0137eb0 con 0x7ffae00b4d60 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffac7fff640 10 monclient: _finish_auth 0 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffac7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffac7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffae5e15640 5 monclient: authenticate success, global_id 4714 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffae5e15640 10 monclient: _renew_subs 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffae5e15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffae5e15640 1 -- 172.21.15.37:0/2782963617 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae0138100 con 0x7ffae00b4d60 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffae5e15640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffae5e15640 10 monclient: _renew_subs 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffae5e15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffae5e15640 1 -- 172.21.15.37:0/2782963617 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae01e3710 con 0x7ffae00b4d60 2024-03-25T18:18:40.820 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.817+0000 7ffae5e15640 1 librados: init done 2024-03-25T18:18:40.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.818+0000 7ffae5e15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:40.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.818+0000 7ffae5e15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:40.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.818+0000 7ffae5e15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:40.821 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:40.818+0000 7ffae5e15640 1 -- 172.21.15.37:0/2782963617 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae00627b0 con 0x7ffae00b4d60 2024-03-25T18:18:41.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.119+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffad40038a0 con 0x7ffae00b4d60 2024-03-25T18:18:41.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.119+0000 7ffac7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:41.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.119+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7ffad4003c70 con 0x7ffae00b4d60 2024-03-25T18:18:41.120 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.119+0000 7ffac7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.119+0000 7ffac7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.119+0000 7ffac7fff640 10 monclient: dump: 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:41.121 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:41.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.119+0000 7ffaddffb640 4 set_mon_vals no callback set 2024-03-25T18:18:41.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.120+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7ffad4003e90 con 0x7ffae00b4d60 2024-03-25T18:18:41.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.121+0000 7ffadf7fe640 10 monclient: get_auth_request con 0x7ffacc06df50 auth_method 0 2024-03-25T18:18:41.122 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.121+0000 7ffac7fff640 10 client.4714.objecter ms_handle_connect 0x7ffacc06df50 2024-03-25T18:18:41.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.124+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7ffad4054920 con 0x7ffae00b4d60 2024-03-25T18:18:41.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.124+0000 7ffac7fff640 10 client.4714.objecter ms_dispatch 0x7ffae0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:41.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.124+0000 7ffac7fff640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:41.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.124+0000 7ffac7fff640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:41.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.124+0000 7ffac7fff640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T18:18:41.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.125+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffad4050430 con 0x7ffae00b4d60 2024-03-25T18:18:41.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.125+0000 7ffac7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:41.126 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.125+0000 7ffac7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:41.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.379+0000 7ffae5e15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:41.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.379+0000 7ffae5e15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:41.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.379+0000 7ffae5e15640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:18:41.380 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.379+0000 7ffae5e15640 1 -- 172.21.15.37:0/2782963617 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffae0145770 con 0x7ffae00b4d60 2024-03-25T18:18:41.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.421+0000 7ffac7fff640 1 -- 172.21.15.37:0/2782963617 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7ffae0145770 con 0x7ffae00b4d60 2024-03-25T18:18:41.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.421+0000 7ffac7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:41.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.421+0000 7ffac7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:41.423 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:18:41.423 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:41.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.423+0000 7ffae5e15640 10 librados: watch_flush enter 2024-03-25T18:18:41.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.423+0000 7ffae5e15640 10 librados: watch_flush exit 2024-03-25T18:18:41.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.423+0000 7ffae5e15640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T18:18:41.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.423+0000 7ffae5e15640 10 client.4714.objecter successfully canceled tick 2024-03-25T18:18:41.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.423+0000 7ffae5e15640 1 -- 172.21.15.37:0/2782963617 >> v1:172.21.15.117:6800/3703993921 conn(0x7ffacc06df50 legacy=0x7ffacc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:41.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.423+0000 7ffae5e15640 10 monclient: shutdown 2024-03-25T18:18:41.425 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.423+0000 7ffae5e15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:41.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.423+0000 7ffae5e15640 1 -- 172.21.15.37:0/2782963617 >> v1:172.21.15.117:6789/0 conn(0x7ffae00b4d60 legacy=0x7ffae01427d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:41.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.423+0000 7ffae5e15640 1 -- 172.21.15.37:0/2782963617 shutdown_connections 2024-03-25T18:18:41.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.423+0000 7ffae5e15640 1 -- 172.21.15.37:0/2782963617 >> 172.21.15.37:0/2782963617 conn(0x7ffae00617a0 msgr2=0x7ffae013c6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:41.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.424+0000 7ffae5e15640 1 -- 172.21.15.37:0/2782963617 shutdown_connections 2024-03-25T18:18:41.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.424+0000 7ffae5e15640 1 -- 172.21.15.37:0/2782963617 wait complete. 2024-03-25T18:18:41.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:41.424+0000 7ffae5e15640 1 librados: shutdown 2024-03-25T18:18:41.436 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:44.946 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:20.291 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:44.953 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:20.299 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:44.962 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:20.275 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:44.965 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:20.278 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:44.974 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:20.319 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:44.979 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:20.324 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:44.980 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:20.325 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:44.985 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:20.298 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:44.998 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:20.343 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:48.242 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:48.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.358+0000 7fe599758640 10 monclient: get_monmap_and_config 2024-03-25T18:18:48.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.358+0000 7fe599758640 10 monclient: build_initial_monmap 2024-03-25T18:18:48.360 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.358+0000 7fe599758640 10 monclient: monmap: 2024-03-25T18:18:48.360 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:48.360 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:48.360 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:18:48.359790+0000 2024-03-25T18:18:48.360 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:18:48.359790+0000 2024-03-25T18:18:48.360 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:48.360 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:48.361 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:48.361 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:18:48.361 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:18:48.361 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:48.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.359+0000 7fe599758640 1 Processor -- start 2024-03-25T18:18:48.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.359+0000 7fe599758640 1 -- start start 2024-03-25T18:18:48.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.359+0000 7fe599758640 10 monclient: init 2024-03-25T18:18:48.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.360+0000 7fe599758640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:48.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.360+0000 7fe599758640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:48.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.360+0000 7fe599758640 10 monclient(hunting): picked mon.noname-b con 0x7fe5940a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:48.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.360+0000 7fe599758640 10 monclient(hunting): picked mon.noname-a con 0x7fe59413f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:48.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.360+0000 7fe599758640 10 monclient(hunting): picked mon.noname-c con 0x7fe5941430c0 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:48.362 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.360+0000 7fe599758640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe594142520 con 0x7fe59413f710 2024-03-25T18:18:48.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.360+0000 7fe599758640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5940b28a0 con 0x7fe5941430c0 2024-03-25T18:18:48.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.360+0000 7fe599758640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5940b2a30 con 0x7fe5940a98f0 2024-03-25T18:18:48.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.360+0000 7fe599758640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:48.363 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.360+0000 7fe599758640 10 monclient(hunting): authenticate will time out at 7483.612975s 2024-03-25T18:18:48.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.367+0000 7fe592ffd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fe5940a98f0 0x7fe5940a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:33494/0 (socket says 172.21.15.37:33494) 2024-03-25T18:18:48.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.367+0000 7fe592ffd640 1 -- 172.21.15.37:0/2622885677 learned_addr learned my addr 172.21.15.37:0/2622885677 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:48.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.376+0000 7fe591ffb640 1 -- 172.21.15.37:0/2622885677 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319868608 0 0) 0x7fe5940b2a30 con 0x7fe5940a98f0 2024-03-25T18:18:48.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.376+0000 7fe591ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:48.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.376+0000 7fe591ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:48.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.376+0000 7fe591ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:48.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.376+0000 7fe591ffb640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:18:48.377 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.376+0000 7fe591ffb640 1 -- 172.21.15.37:0/2622885677 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe57c003520 con 0x7fe5940a98f0 2024-03-25T18:18:48.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.387+0000 7fe591ffb640 1 -- 172.21.15.37:0/2622885677 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3709785527 0 0) 0x7fe5940b28a0 con 0x7fe5941430c0 2024-03-25T18:18:48.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.387+0000 7fe591ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:48.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.387+0000 7fe591ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:48.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.387+0000 7fe591ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:48.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.387+0000 7fe591ffb640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:18:48.388 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.387+0000 7fe591ffb640 1 -- 172.21.15.37:0/2622885677 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5940b2a30 con 0x7fe5941430c0 2024-03-25T18:18:48.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 1 -- 172.21.15.37:0/2622885677 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2385206440 0 0) 0x7fe57c003520 con 0x7fe5940a98f0 2024-03-25T18:18:48.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 1 -- 172.21.15.37:0/2622885677 >> v1:172.21.15.117:6789/0 conn(0x7fe5941430c0 legacy=0x7fe59414dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:48.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 1 -- 172.21.15.37:0/2622885677 >> v1:172.21.15.37:6789/0 conn(0x7fe59413f710 legacy=0x7fe5941429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:48.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:48.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 1 monclient: found mon.noname-b 2024-03-25T18:18:48.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:48.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:18:48.392 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 1 -- 172.21.15.37:0/2622885677 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5940b44c0 con 0x7fe5940a98f0 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 1 -- 172.21.15.37:0/2622885677 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe584001690 con 0x7fe5940a98f0 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe599758640 5 monclient: authenticate success, global_id 4715 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe599758640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe591ffb640 10 monclient: dump: 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.393 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:48.394 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:48.394 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:48.394 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:48.394 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:48.394 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:48.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.390+0000 7fe599758640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:48.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.421+0000 7fe591ffb640 1 -- 172.21.15.37:0/2622885677 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe5840017f0 con 0x7fe5940a98f0 2024-03-25T18:18:48.422 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.421+0000 7fe591ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.421+0000 7fe591ffb640 1 -- 172.21.15.37:0/2622885677 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe584001c00 con 0x7fe5940a98f0 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.421+0000 7fe599758640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.421+0000 7fe599758640 4 set_mon_vals no callback set 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.421+0000 7fe591ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.421+0000 7fe591ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.421+0000 7fe591ffb640 10 monclient: dump: 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:48.423 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:48.424 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:48.424 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:48.424 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:48.424 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:48.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.422+0000 7fe599758640 10 monclient: shutdown 2024-03-25T18:18:48.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.422+0000 7fe599758640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:48.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.422+0000 7fe599758640 1 -- 172.21.15.37:0/2622885677 >> v1:172.21.15.37:6790/0 conn(0x7fe5940a98f0 legacy=0x7fe5940a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:48.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.422+0000 7fe599758640 1 -- 172.21.15.37:0/2622885677 shutdown_connections 2024-03-25T18:18:48.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.422+0000 7fe599758640 1 -- 172.21.15.37:0/2622885677 >> 172.21.15.37:0/2622885677 conn(0x7fe5940b7ea0 msgr2=0x7fe5940b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:48.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.422+0000 7fe599758640 1 -- 172.21.15.37:0/2622885677 shutdown_connections 2024-03-25T18:18:48.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.422+0000 7fe599758640 1 -- 172.21.15.37:0/2622885677 wait complete. 2024-03-25T18:18:48.426 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.423+0000 7fe599758640 10 monclient: build_initial_monmap 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.423+0000 7fe599758640 10 monclient: monmap: 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.423+0000 7fe599758640 1 librados: starting msgr at 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.423+0000 7fe599758640 1 librados: starting objecter 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.423+0000 7fe599758640 1 Processor -- start 2024-03-25T18:18:48.427 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.423+0000 7fe599758640 1 -- start start 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.423+0000 7fe599758640 1 librados: setting wanted keys 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.423+0000 7fe599758640 1 librados: calling monclient init 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.423+0000 7fe599758640 10 monclient: init 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.424+0000 7fe599758640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.424+0000 7fe599758640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.424+0000 7fe599758640 10 monclient(hunting): picked mon.noname-b con 0x7fe5940a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.424+0000 7fe599758640 10 monclient(hunting): picked mon.noname-c con 0x7fe59413f710 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.424+0000 7fe599758640 10 monclient(hunting): picked mon.noname-a con 0x7fe5941430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.424+0000 7fe599758640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5941db7d0 con 0x7fe5941430c0 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.424+0000 7fe599758640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5940b1420 con 0x7fe5940a98f0 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.424+0000 7fe599758640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe5940b15b0 con 0x7fe59413f710 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.424+0000 7fe599758640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:48.428 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.424+0000 7fe599758640 10 monclient(hunting): authenticate will time out at 7483.676871s 2024-03-25T18:18:48.435 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.434+0000 7fe5927fc640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7fe59413f710 0x7fe5940b01e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:33524/0 (socket says 172.21.15.37:33524) 2024-03-25T18:18:48.435 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.434+0000 7fe5927fc640 1 -- 172.21.15.37:0/3781902532 learned_addr learned my addr 172.21.15.37:0/3781902532 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:48.438 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.437+0000 7fe57b7fe640 10 client.?.objecter ms_handle_connect 0x7fe5940a98f0 2024-03-25T18:18:48.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.437+0000 7fe57b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:48.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.437+0000 7fe57b7fe640 10 client.?.objecter ms_handle_connect 0x7fe59413f710 2024-03-25T18:18:48.439 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.437+0000 7fe57b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:48.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.439+0000 7fe57b7fe640 10 client.?.objecter ms_handle_connect 0x7fe5941430c0 2024-03-25T18:18:48.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.440+0000 7fe57b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:48.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.446+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623935663 0 0) 0x7fe5940b1420 con 0x7fe5940a98f0 2024-03-25T18:18:48.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.446+0000 7fe57b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:48.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.446+0000 7fe57b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:48.447 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.446+0000 7fe57b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:48.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.446+0000 7fe57b7fe640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:18:48.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.446+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe580004eb0 con 0x7fe5940a98f0 2024-03-25T18:18:48.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.453+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002654431 0 0) 0x7fe5940b15b0 con 0x7fe59413f710 2024-03-25T18:18:48.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.453+0000 7fe57b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:48.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.453+0000 7fe57b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:48.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.453+0000 7fe57b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:48.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.453+0000 7fe57b7fe640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:18:48.454 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.453+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5940b1420 con 0x7fe59413f710 2024-03-25T18:18:48.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.458+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435556777 0 0) 0x7fe5941db7d0 con 0x7fe5941430c0 2024-03-25T18:18:48.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.458+0000 7fe57b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:48.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.458+0000 7fe57b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:48.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.458+0000 7fe57b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:48.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.458+0000 7fe57b7fe640 10 monclient(hunting): my global_id is 4734 2024-03-25T18:18:48.459 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.458+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe5940b15b0 con 0x7fe5941430c0 2024-03-25T18:18:48.471 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.470+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2672939965 0 0) 0x7fe5940b1420 con 0x7fe59413f710 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.470+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5941db7d0 con 0x7fe59413f710 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.470+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe5880019b0 con 0x7fe59413f710 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.470+0000 7fe57b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.470+0000 7fe57b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.470+0000 7fe57b7fe640 10 monclient(hunting): dump: 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:48.472 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:48.473 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:48.473 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:48.473 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.470+0000 7fe57b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:48.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.474+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2554837980 0 0) 0x7fe580004eb0 con 0x7fe5940a98f0 2024-03-25T18:18:48.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.474+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe5940b1420 con 0x7fe5940a98f0 2024-03-25T18:18:48.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.474+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe584001b00 con 0x7fe5940a98f0 2024-03-25T18:18:48.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.474+0000 7fe57b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:48.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.474+0000 7fe57b7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:48.475 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.474+0000 7fe57b7fe640 10 monclient(hunting): dump: 2024-03-25T18:18:48.475 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:48.475 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:48.475 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.476 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.476 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:48.476 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:48.476 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:48.476 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:48.476 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:48.476 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:48.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.477+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2005054883 0 0) 0x7fe5940b15b0 con 0x7fe5941430c0 2024-03-25T18:18:48.478 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.477+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe580004eb0 con 0x7fe5941430c0 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.477+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe58c001bd0 con 0x7fe5941430c0 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.477+0000 7fe57b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.477+0000 7fe57b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.477+0000 7fe57b7fe640 10 monclient(hunting): dump: 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:48.479 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:48.482 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 60068421 0 0) 0x7fe5941db7d0 con 0x7fe59413f710 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 >> v1:172.21.15.117:6789/0 conn(0x7fe5940a98f0 legacy=0x7fe5941daa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 >> v1:172.21.15.37:6789/0 conn(0x7fe5941430c0 legacy=0x7fe5941d4350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe57b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe57b7fe640 1 monclient: found mon.c 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe57b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe57b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5940b1740 con 0x7fe59413f710 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe57b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe57b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe57b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe599758640 5 monclient: authenticate success, global_id 4718 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe599758640 10 monclient: _renew_subs 2024-03-25T18:18:48.483 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe599758640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:48.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe599758640 1 -- 172.21.15.37:0/3781902532 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5941db1a0 con 0x7fe59413f710 2024-03-25T18:18:48.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe599758640 10 client.4718.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:48.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe599758640 10 monclient: _renew_subs 2024-03-25T18:18:48.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe599758640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:48.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe599758640 1 -- 172.21.15.37:0/3781902532 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5940b08f0 con 0x7fe59413f710 2024-03-25T18:18:48.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.481+0000 7fe599758640 1 librados: init done 2024-03-25T18:18:48.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.482+0000 7fe599758640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:48.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.482+0000 7fe599758640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:48.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.482+0000 7fe599758640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:48.484 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.482+0000 7fe599758640 1 -- 172.21.15.37:0/3781902532 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5940627b0 con 0x7fe59413f710 2024-03-25T18:18:48.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.505+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe588003610 con 0x7fe59413f710 2024-03-25T18:18:48.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.505+0000 7fe57b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:48.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.505+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7fe588003920 con 0x7fe59413f710 2024-03-25T18:18:48.506 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.505+0000 7fe57b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.505+0000 7fe57b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.505+0000 7fe57b7fe640 10 monclient: dump: 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:48.507 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.505+0000 7fe5917fa640 4 set_mon_vals no callback set 2024-03-25T18:18:48.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.506+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7fe588003b40 con 0x7fe59413f710 2024-03-25T18:18:48.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.507+0000 7fe592ffd640 10 monclient: get_auth_request con 0x7fe58006e110 auth_method 0 2024-03-25T18:18:48.508 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.507+0000 7fe57b7fe640 10 client.4718.objecter ms_handle_connect 0x7fe58006e110 2024-03-25T18:18:48.510 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.509+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7fe5880547e0 con 0x7fe59413f710 2024-03-25T18:18:48.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.509+0000 7fe57b7fe640 10 client.4718.objecter ms_dispatch 0x7fe594000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:48.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.509+0000 7fe57b7fe640 3 client.4718.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:48.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.509+0000 7fe57b7fe640 3 client.4718.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:48.511 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.510+0000 7fe57b7fe640 20 client.4718.objecter dump_active .. 0 homeless 2024-03-25T18:18:48.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.811+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe5880502f0 con 0x7fe59413f710 2024-03-25T18:18:48.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.811+0000 7fe57b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:48.812 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:48.811+0000 7fe57b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:49.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.063+0000 7fe599758640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:49.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.064+0000 7fe599758640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:49.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.064+0000 7fe599758640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:49.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.064+0000 7fe599758640 1 -- 172.21.15.37:0/3781902532 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe5940abae0 con 0x7fe59413f710 2024-03-25T18:18:49.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.092+0000 7fe57b7fe640 1 -- 172.21.15.37:0/3781902532 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7fe588052220 con 0x7fe59413f710 2024-03-25T18:18:49.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.092+0000 7fe57b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:49.094 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.092+0000 7fe57b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:49.094 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:18:49.094 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:49.096 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 10 librados: watch_flush enter 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 10 librados: watch_flush exit 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 20 client.4718.objecter shutdown clearing up homeless session... 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 10 client.4718.objecter successfully canceled tick 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 1 -- 172.21.15.37:0/3781902532 >> v1:172.21.15.117:6800/3703993921 conn(0x7fe58006e110 legacy=0x7fe5800705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 10 monclient: shutdown 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 1 -- 172.21.15.37:0/3781902532 >> v1:172.21.15.37:6790/0 conn(0x7fe59413f710 legacy=0x7fe5940b01e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 1 -- 172.21.15.37:0/3781902532 shutdown_connections 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 1 -- 172.21.15.37:0/3781902532 >> 172.21.15.37:0/3781902532 conn(0x7fe5940b7ea0 msgr2=0x7fe59413c9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 1 -- 172.21.15.37:0/3781902532 shutdown_connections 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 1 -- 172.21.15.37:0/3781902532 wait complete. 2024-03-25T18:18:49.097 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:49.095+0000 7fe599758640 1 librados: shutdown 2024-03-25T18:18:49.109 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:55.913 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:56.029 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.028+0000 7f50c77cf640 10 monclient: get_monmap_and_config 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.028+0000 7f50c77cf640 10 monclient: build_initial_monmap 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.028+0000 7f50c77cf640 10 monclient: monmap: 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:18:56.029764+0000 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:18:56.029764+0000 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:18:56.030 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:56.031 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.029+0000 7f50c77cf640 1 Processor -- start 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.029+0000 7f50c77cf640 1 -- start start 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.029+0000 7f50c77cf640 10 monclient: init 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.030+0000 7f50c77cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.030+0000 7f50c77cf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.030+0000 7f50c77cf640 10 monclient(hunting): picked mon.noname-a con 0x7f50c00b4d60 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.030+0000 7f50c77cf640 10 monclient(hunting): picked mon.noname-c con 0x7f50c00b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.030+0000 7f50c77cf640 10 monclient(hunting): picked mon.noname-b con 0x7f50c014be90 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.030+0000 7f50c77cf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c00af940 con 0x7f50c00b4d60 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.030+0000 7f50c77cf640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c00b0a70 con 0x7f50c00b1730 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.030+0000 7f50c77cf640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c00b0d50 con 0x7f50c014be90 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.030+0000 7f50c77cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:56.032 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.030+0000 7f50c77cf640 10 monclient(hunting): authenticate will time out at 7491.282980s 2024-03-25T18:18:56.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.037+0000 7f50c5d6c640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f50c014be90 0x7f50c014e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:38122/0 (socket says 172.21.15.37:38122) 2024-03-25T18:18:56.038 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.037+0000 7f50c5d6c640 1 -- 172.21.15.37:0/1975544213 learned_addr learned my addr 172.21.15.37:0/1975544213 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:56.045 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.044+0000 7f50b7fff640 1 -- 172.21.15.37:0/1975544213 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2100949638 0 0) 0x7f50c00b0d50 con 0x7f50c014be90 2024-03-25T18:18:56.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.044+0000 7f50b7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:18:56.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.044+0000 7f50b7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:56.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.044+0000 7f50b7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:56.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.044+0000 7f50b7fff640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:18:56.046 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.044+0000 7f50b7fff640 1 -- 172.21.15.37:0/1975544213 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50ac003520 con 0x7f50c014be90 2024-03-25T18:18:56.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 1 -- 172.21.15.37:0/1975544213 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2590443833 0 0) 0x7f50ac003520 con 0x7f50c014be90 2024-03-25T18:18:56.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 1 -- 172.21.15.37:0/1975544213 >> v1:172.21.15.117:6789/0 conn(0x7f50c00b1730 legacy=0x7f50c00b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:56.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 1 -- 172.21.15.37:0/1975544213 >> v1:172.21.15.37:6789/0 conn(0x7f50c00b4d60 legacy=0x7f50c013f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:56.056 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 1 monclient: found mon.noname-b 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 1 -- 172.21.15.37:0/1975544213 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c00b1030 con 0x7f50c014be90 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 10 monclient: _finish_auth 0 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 1 -- 172.21.15.37:0/1975544213 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f50b8001a10 con 0x7f50c014be90 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50b7fff640 10 monclient: dump: 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:56.057 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:56.058 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:56.058 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:56.058 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:56.058 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:56.058 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:56.058 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:56.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50c77cf640 5 monclient: authenticate success, global_id 4721 2024-03-25T18:18:56.058 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.055+0000 7f50c77cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.060+0000 7f50b7fff640 1 -- 172.21.15.37:0/1975544213 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50b8001be0 con 0x7f50c014be90 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.060+0000 7f50b7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.060+0000 7f50b7fff640 1 -- 172.21.15.37:0/1975544213 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f50b8002030 con 0x7f50c014be90 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.060+0000 7f50b7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.060+0000 7f50b7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.060+0000 7f50b7fff640 10 monclient: dump: 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:56.061 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:56.062 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:56.062 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:56.062 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:56.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.060+0000 7f50c77cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:56.062 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.060+0000 7f50c77cf640 4 set_mon_vals no callback set 2024-03-25T18:18:56.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.060+0000 7f50c77cf640 10 monclient: shutdown 2024-03-25T18:18:56.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.060+0000 7f50c77cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:56.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.060+0000 7f50c77cf640 1 -- 172.21.15.37:0/1975544213 >> v1:172.21.15.37:6790/0 conn(0x7f50c014be90 legacy=0x7f50c014e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:56.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.061+0000 7f50c77cf640 1 -- 172.21.15.37:0/1975544213 shutdown_connections 2024-03-25T18:18:56.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.061+0000 7f50c77cf640 1 -- 172.21.15.37:0/1975544213 >> 172.21.15.37:0/1975544213 conn(0x7f50c00617a0 msgr2=0x7f50c0061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:56.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.061+0000 7f50c77cf640 1 -- 172.21.15.37:0/1975544213 shutdown_connections 2024-03-25T18:18:56.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.061+0000 7f50c77cf640 1 -- 172.21.15.37:0/1975544213 wait complete. 2024-03-25T18:18:56.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.061+0000 7f50c77cf640 10 monclient: build_initial_monmap 2024-03-25T18:18:56.063 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.061+0000 7f50c77cf640 10 monclient: monmap: 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.062+0000 7f50c77cf640 1 librados: starting msgr at 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.062+0000 7f50c77cf640 1 librados: starting objecter 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.062+0000 7f50c77cf640 1 Processor -- start 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.062+0000 7f50c77cf640 1 -- start start 2024-03-25T18:18:56.064 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.062+0000 7f50c77cf640 1 librados: setting wanted keys 2024-03-25T18:18:56.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.062+0000 7f50c77cf640 1 librados: calling monclient init 2024-03-25T18:18:56.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.062+0000 7f50c77cf640 10 monclient: init 2024-03-25T18:18:56.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.062+0000 7f50c77cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:56.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.063+0000 7f50c77cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:56.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.063+0000 7f50c77cf640 10 monclient(hunting): picked mon.noname-a con 0x7f50c00b1730 addr v1:172.21.15.37:6789/0 2024-03-25T18:18:56.065 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.063+0000 7f50c77cf640 10 monclient(hunting): picked mon.noname-c con 0x7f50c00b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:18:56.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.063+0000 7f50c77cf640 10 monclient(hunting): picked mon.noname-b con 0x7f50c014be90 addr v1:172.21.15.117:6789/0 2024-03-25T18:18:56.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.063+0000 7f50c77cf640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c0140380 con 0x7f50c00b1730 2024-03-25T18:18:56.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.063+0000 7f50c77cf640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c013bde0 con 0x7f50c014be90 2024-03-25T18:18:56.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.063+0000 7f50c77cf640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50c013bf70 con 0x7f50c00b4d60 2024-03-25T18:18:56.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.063+0000 7f50c77cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:56.066 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.063+0000 7f50c77cf640 10 monclient(hunting): authenticate will time out at 7491.315381s 2024-03-25T18:18:56.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.074+0000 7f50c5d6c640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f50c014be90 0x7f50c013ac60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:37954/0 (socket says 172.21.15.37:37954) 2024-03-25T18:18:56.075 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.074+0000 7f50c5d6c640 1 -- 172.21.15.37:0/3326462004 learned_addr learned my addr 172.21.15.37:0/3326462004 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:18:56.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.078+0000 7f50b5ffb640 10 client.?.objecter ms_handle_connect 0x7f50c00b4d60 2024-03-25T18:18:56.079 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.078+0000 7f50b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:56.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.080+0000 7f50b5ffb640 10 client.?.objecter ms_handle_connect 0x7f50c014be90 2024-03-25T18:18:56.081 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.080+0000 7f50b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:56.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.082+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1646467211 0 0) 0x7f50b0001090 con 0x7f50c00b4d60 2024-03-25T18:18:56.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.082+0000 7f50b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:56.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.082+0000 7f50b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:56.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.082+0000 7f50b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:56.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.082+0000 7f50b5ffb640 10 monclient(hunting): my global_id is 4724 2024-03-25T18:18:56.083 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.082+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50a4004eb0 con 0x7f50c00b4d60 2024-03-25T18:18:56.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.087+0000 7f50b5ffb640 10 client.?.objecter ms_handle_connect 0x7f50c00b1730 2024-03-25T18:18:56.088 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.087+0000 7f50b5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:56.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.097+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035345033 0 0) 0x7f50c013bde0 con 0x7f50c014be90 2024-03-25T18:18:56.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.097+0000 7f50b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:56.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.097+0000 7f50b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:56.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.097+0000 7f50b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:56.098 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.097+0000 7f50b5ffb640 10 monclient(hunting): my global_id is 4723 2024-03-25T18:18:56.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.097+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50b0001090 con 0x7f50c014be90 2024-03-25T18:18:56.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.097+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732095629 0 0) 0x7f50c0140380 con 0x7f50c00b1730 2024-03-25T18:18:56.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.098+0000 7f50b5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:56.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.098+0000 7f50b5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:56.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.098+0000 7f50b5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:56.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.098+0000 7f50b5ffb640 10 monclient(hunting): my global_id is 4737 2024-03-25T18:18:56.099 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.098+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f50c013bde0 con 0x7f50c00b1730 2024-03-25T18:18:56.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.104+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3311417206 0 0) 0x7f50a4004eb0 con 0x7f50c00b4d60 2024-03-25T18:18:56.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.104+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f50c0140380 con 0x7f50c00b4d60 2024-03-25T18:18:56.105 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.104+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f50b0001860 con 0x7f50c00b4d60 2024-03-25T18:18:56.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.104+0000 7f50b5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:56.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.104+0000 7f50b5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:56.106 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.104+0000 7f50b5ffb640 10 monclient(hunting): dump: 2024-03-25T18:18:56.106 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:56.106 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:56.106 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:56.106 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:56.106 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:56.106 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:56.106 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:56.106 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:56.107 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:56.107 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:56.107 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.104+0000 7f50b5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:56.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1455899592 0 0) 0x7f50c0140380 con 0x7f50c00b4d60 2024-03-25T18:18:56.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 >> v1:172.21.15.117:6789/0 conn(0x7f50c014be90 legacy=0x7f50c013ac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:56.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 >> v1:172.21.15.37:6789/0 conn(0x7f50c00b1730 legacy=0x7f50c0141930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:56.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50b5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:56.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50b5ffb640 1 monclient: found mon.c 2024-03-25T18:18:56.113 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50b5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50b5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50c0142040 con 0x7f50c00b4d60 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50b5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50b5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50b5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.111+0000 7f50c77cf640 5 monclient: authenticate success, global_id 4724 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.112+0000 7f50c77cf640 10 monclient: _renew_subs 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.112+0000 7f50c77cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.112+0000 7f50c77cf640 1 -- 172.21.15.37:0/3326462004 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50c01d7fb0 con 0x7f50c00b4d60 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.112+0000 7f50c77cf640 10 client.4724.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.112+0000 7f50c77cf640 10 monclient: _renew_subs 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.112+0000 7f50c77cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.112+0000 7f50c77cf640 1 -- 172.21.15.37:0/3326462004 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f50c0140d00 con 0x7f50c00b4d60 2024-03-25T18:18:56.114 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.112+0000 7f50c77cf640 1 librados: init done 2024-03-25T18:18:56.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.113+0000 7f50c77cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:56.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.113+0000 7f50c77cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:56.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.113+0000 7f50c77cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:56.115 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.113+0000 7f50c77cf640 1 -- 172.21.15.37:0/3326462004 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50c00627b0 con 0x7f50c00b4d60 2024-03-25T18:18:56.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.439+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f50b0001d70 con 0x7f50c00b4d60 2024-03-25T18:18:56.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.439+0000 7f50b5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:56.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.439+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f50b0003790 con 0x7f50c00b4d60 2024-03-25T18:18:56.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.439+0000 7f50b5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:56.440 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.439+0000 7f50b5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.439+0000 7f50b5ffb640 10 monclient: dump: 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:18:56.441 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.439+0000 7f50b77fe640 4 set_mon_vals no callback set 2024-03-25T18:18:56.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.440+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f50b0019820 con 0x7f50c00b4d60 2024-03-25T18:18:56.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.441+0000 7f50c556b640 10 monclient: get_auth_request con 0x7f50a406ddf0 auth_method 0 2024-03-25T18:18:56.442 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.441+0000 7f50b5ffb640 10 client.4724.objecter ms_handle_connect 0x7f50a406ddf0 2024-03-25T18:18:56.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.447+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7f50b00535a0 con 0x7f50c00b4d60 2024-03-25T18:18:56.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.447+0000 7f50b5ffb640 10 client.4724.objecter ms_dispatch 0x7f50c0000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:18:56.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.447+0000 7f50b5ffb640 3 client.4724.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:18:56.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.447+0000 7f50b5ffb640 3 client.4724.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:18:56.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.447+0000 7f50b5ffb640 20 client.4724.objecter dump_active .. 0 homeless 2024-03-25T18:18:56.448 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.447+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f50b004eee0 con 0x7f50c00b4d60 2024-03-25T18:18:56.449 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.447+0000 7f50b5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:56.449 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.447+0000 7f50b5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:56.514 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:31.859 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:56.559 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:31.871 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:56.561 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:31.874 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:56.565 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:31.910 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:56.604 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:31.950 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:56.606 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:31.919 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:56.630 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:31.975 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:56.640 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:31.985 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:56.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.673+0000 7f50c77cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:56.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.673+0000 7f50c77cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:56.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.673+0000 7f50c77cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:18:56.675 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.673+0000 7f50c77cf640 1 -- 172.21.15.37:0/3326462004 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f50c01458b0 con 0x7f50c00b4d60 2024-03-25T18:18:56.677 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:32.023 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:56.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.705+0000 7f50b5ffb640 1 -- 172.21.15.37:0/3326462004 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7f50b0050fe0 con 0x7f50c00b4d60 2024-03-25T18:18:56.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.705+0000 7f50b5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:56.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.705+0000 7f50b5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:56.706 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:18:56.707 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:56.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 10 librados: watch_flush enter 2024-03-25T18:18:56.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 10 librados: watch_flush exit 2024-03-25T18:18:56.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 20 client.4724.objecter shutdown clearing up homeless session... 2024-03-25T18:18:56.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 10 client.4724.objecter successfully canceled tick 2024-03-25T18:18:56.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 1 -- 172.21.15.37:0/3326462004 >> v1:172.21.15.117:6800/3703993921 conn(0x7f50a406ddf0 legacy=0x7f50a40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:56.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 10 monclient: shutdown 2024-03-25T18:18:56.709 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 1 -- 172.21.15.37:0/3326462004 >> v1:172.21.15.37:6790/0 conn(0x7f50c00b4d60 legacy=0x7f50c01d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 1 -- 172.21.15.37:0/3326462004 shutdown_connections 2024-03-25T18:18:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 1 -- 172.21.15.37:0/3326462004 >> 172.21.15.37:0/3326462004 conn(0x7f50c00617a0 msgr2=0x7f50c00b7c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 1 -- 172.21.15.37:0/3326462004 shutdown_connections 2024-03-25T18:18:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 1 -- 172.21.15.37:0/3326462004 wait complete. 2024-03-25T18:18:56.710 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:18:56.707+0000 7f50c77cf640 1 librados: shutdown 2024-03-25T18:18:56.719 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:03.526 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:03.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.643+0000 7f6cc46f1640 10 monclient: get_monmap_and_config 2024-03-25T18:19:03.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.643+0000 7f6cc46f1640 10 monclient: build_initial_monmap 2024-03-25T18:19:03.644 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.643+0000 7f6cc46f1640 10 monclient: monmap: 2024-03-25T18:19:03.644 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:19:03.645 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:03.645 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:19:03.644301+0000 2024-03-25T18:19:03.645 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:19:03.644301+0000 2024-03-25T18:19:03.645 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:03.645 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:03.645 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:19:03.645 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:19:03.645 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:19:03.645 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:03.646 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.644+0000 7f6cc46f1640 1 Processor -- start 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.644+0000 7f6cc46f1640 1 -- start start 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.644+0000 7f6cc46f1640 10 monclient: init 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.645+0000 7f6cc46f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.645+0000 7f6cc46f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.645+0000 7f6cc46f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6cbc0a98f0 addr v1:172.21.15.37:6790/0 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.645+0000 7f6cc46f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6cbc13f710 addr v1:172.21.15.117:6789/0 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.645+0000 7f6cc46f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6cbc1430c0 addr v1:172.21.15.37:6789/0 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.645+0000 7f6cc46f1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cbc142520 con 0x7f6cbc1430c0 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.645+0000 7f6cc46f1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cbc0b28a0 con 0x7f6cbc13f710 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.645+0000 7f6cc46f1640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cbc0b2a30 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.645+0000 7f6cc46f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:03.647 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.645+0000 7f6cc46f1640 10 monclient(hunting): authenticate will time out at 7498.897621s 2024-03-25T18:19:03.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.656+0000 7f6cc248d640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f6cbc0a98f0 0x7f6cbc0a9cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:39374/0 (socket says 172.21.15.37:39374) 2024-03-25T18:19:03.657 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.656+0000 7f6cc248d640 1 -- 172.21.15.37:0/1636732250 learned_addr learned my addr 172.21.15.37:0/1636732250 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:19:03.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.672+0000 7f6cc148b640 1 -- 172.21.15.37:0/1636732250 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3824336211 0 0) 0x7f6cbc0b2a30 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.672+0000 7f6cc148b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:03.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.672+0000 7f6cc148b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:03.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.672+0000 7f6cc148b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:03.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.672+0000 7f6cc148b640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:19:03.673 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.672+0000 7f6cc148b640 1 -- 172.21.15.37:0/1636732250 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ca8003520 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.675+0000 7f6cc148b640 1 -- 172.21.15.37:0/1636732250 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3025714976 0 0) 0x7f6cbc0b28a0 con 0x7f6cbc13f710 2024-03-25T18:19:03.676 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.675+0000 7f6cc148b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:03.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.675+0000 7f6cc148b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:03.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.675+0000 7f6cc148b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:03.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.675+0000 7f6cc148b640 10 monclient(hunting): my global_id is 4726 2024-03-25T18:19:03.677 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.675+0000 7f6cc148b640 1 -- 172.21.15.37:0/1636732250 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cbc0b2a30 con 0x7f6cbc13f710 2024-03-25T18:19:03.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.685+0000 7f6cc148b640 1 -- 172.21.15.37:0/1636732250 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3621191210 0 0) 0x7f6ca8003520 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.685+0000 7f6cc148b640 1 -- 172.21.15.37:0/1636732250 >> v1:172.21.15.117:6789/0 conn(0x7f6cbc13f710 legacy=0x7f6cbc1429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:03.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.685+0000 7f6cc148b640 1 -- 172.21.15.37:0/1636732250 >> v1:172.21.15.37:6789/0 conn(0x7f6cbc1430c0 legacy=0x7f6cbc14dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:03.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 10 monclient: _finish_hunting 0 2024-03-25T18:19:03.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 1 monclient: found mon.noname-b 2024-03-25T18:19:03.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:03.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.37:6790/0 2024-03-25T18:19:03.687 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 1 -- 172.21.15.37:0/1636732250 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc0b44c0 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 10 monclient: _finish_auth 0 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 10 monclient: _check_auth_tickets 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 1 -- 172.21.15.37:0/1636732250 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6cb8001690 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc46f1640 5 monclient: authenticate success, global_id 4727 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc148b640 10 monclient: dump: 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:03.688 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:03.689 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:03.689 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:19:03.689 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:19:03.689 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:19:03.689 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:03.689 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.686+0000 7f6cc46f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:03.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.696+0000 7f6cc148b640 1 -- 172.21.15.37:0/1636732250 <== mon.2 v1:172.21.15.37:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cb80017f0 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.696+0000 7f6cc148b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:03.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.696+0000 7f6cc148b640 1 -- 172.21.15.37:0/1636732250 <== mon.2 v1:172.21.15.37:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6cb8001c00 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.696+0000 7f6cc46f1640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:03.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.696+0000 7f6cc46f1640 4 set_mon_vals no callback set 2024-03-25T18:19:03.697 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.696+0000 7f6cc148b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.696+0000 7f6cc148b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.696+0000 7f6cc148b640 10 monclient: dump: 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:19:03.698 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:03.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.696+0000 7f6cc46f1640 10 monclient: shutdown 2024-03-25T18:19:03.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.696+0000 7f6cc46f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:03.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.696+0000 7f6cc46f1640 1 -- 172.21.15.37:0/1636732250 >> v1:172.21.15.37:6790/0 conn(0x7f6cbc0a98f0 legacy=0x7f6cbc0a9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:03.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.697+0000 7f6cc46f1640 1 -- 172.21.15.37:0/1636732250 shutdown_connections 2024-03-25T18:19:03.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.697+0000 7f6cc46f1640 1 -- 172.21.15.37:0/1636732250 >> 172.21.15.37:0/1636732250 conn(0x7f6cbc0b7ea0 msgr2=0x7f6cbc0b82b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:03.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.697+0000 7f6cc46f1640 1 -- 172.21.15.37:0/1636732250 shutdown_connections 2024-03-25T18:19:03.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.697+0000 7f6cc46f1640 1 -- 172.21.15.37:0/1636732250 wait complete. 2024-03-25T18:19:03.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.698+0000 7f6cc46f1640 10 monclient: build_initial_monmap 2024-03-25T18:19:03.700 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.698+0000 7f6cc46f1640 10 monclient: monmap: 2024-03-25T18:19:03.700 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:19:03.700 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.698+0000 7f6cc46f1640 1 librados: starting msgr at 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.698+0000 7f6cc46f1640 1 librados: starting objecter 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.698+0000 7f6cc46f1640 1 Processor -- start 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.698+0000 7f6cc46f1640 1 -- start start 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.698+0000 7f6cc46f1640 1 librados: setting wanted keys 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.698+0000 7f6cc46f1640 1 librados: calling monclient init 2024-03-25T18:19:03.701 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.698+0000 7f6cc46f1640 10 monclient: init 2024-03-25T18:19:03.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.699+0000 7f6cc46f1640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:03.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.699+0000 7f6cc46f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:03.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.699+0000 7f6cc46f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6cbc0a98f0 addr v1:172.21.15.117:6789/0 2024-03-25T18:19:03.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.699+0000 7f6cc46f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6cbc13f710 addr v1:172.21.15.37:6789/0 2024-03-25T18:19:03.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.700+0000 7f6cc46f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6cbc1430c0 addr v1:172.21.15.37:6790/0 2024-03-25T18:19:03.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.700+0000 7f6cc46f1640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cbc13be70 con 0x7f6cbc13f710 2024-03-25T18:19:03.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.700+0000 7f6cc46f1640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cbc131660 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.700+0000 7f6cc46f1640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cbc1317f0 con 0x7f6cbc1430c0 2024-03-25T18:19:03.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.700+0000 7f6cc46f1640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:03.703 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.700+0000 7f6cc46f1640 10 monclient(hunting): authenticate will time out at 7498.952523s 2024-03-25T18:19:03.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.705+0000 7f6cc248d640 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f6cbc0a98f0 0x7f6cbc130f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.37:50052/0 (socket says 172.21.15.37:50052) 2024-03-25T18:19:03.706 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.705+0000 7f6cc248d640 1 -- 172.21.15.37:0/3333320801 learned_addr learned my addr 172.21.15.37:0/3333320801 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:19:03.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.706+0000 7f6cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f6cbc0a98f0 2024-03-25T18:19:03.707 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.706+0000 7f6cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:03.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.711+0000 7f6cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f6cbc13f710 2024-03-25T18:19:03.712 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.711+0000 7f6cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:03.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.714+0000 7f6cb2ffd640 10 client.?.objecter ms_handle_connect 0x7f6cbc1430c0 2024-03-25T18:19:03.715 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.714+0000 7f6cb2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:03.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.725+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639569444 0 0) 0x7f6cbc131660 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.725+0000 7f6cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:03.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.725+0000 7f6cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:03.726 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.725+0000 7f6cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:03.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.725+0000 7f6cb2ffd640 10 monclient(hunting): my global_id is 4729 2024-03-25T18:19:03.727 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.725+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ca0004eb0 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.733+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226226164 0 0) 0x7f6cbc1317f0 con 0x7f6cbc1430c0 2024-03-25T18:19:03.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.733+0000 7f6cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:03.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.733+0000 7f6cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:03.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.733+0000 7f6cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:03.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.733+0000 7f6cb2ffd640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:19:03.734 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.733+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cbc131660 con 0x7f6cbc1430c0 2024-03-25T18:19:03.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.735+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 371649559 0 0) 0x7f6cbc13be70 con 0x7f6cbc13f710 2024-03-25T18:19:03.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.735+0000 7f6cb2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:03.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.735+0000 7f6cb2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:03.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.735+0000 7f6cb2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:03.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.735+0000 7f6cb2ffd640 10 monclient(hunting): my global_id is 4743 2024-03-25T18:19:03.736 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.735+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cbc1317f0 con 0x7f6cbc13f710 2024-03-25T18:19:03.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.738+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3926399396 0 0) 0x7f6ca0004eb0 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.739+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6cbc13be70 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.739+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6cb8001bc0 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.739+0000 7f6cb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:03.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.739+0000 7f6cb2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:03.740 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.739+0000 7f6cb2ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:03.740 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:19:03.740 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:03.740 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:03.740 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:03.740 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:03.741 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:03.741 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:19:03.741 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:19:03.741 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:19:03.741 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:03.741 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.739+0000 7f6cb2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:03.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.741+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 556092169 0 0) 0x7f6cbc131660 con 0x7f6cbc1430c0 2024-03-25T18:19:03.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.741+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ca0004eb0 con 0x7f6cbc1430c0 2024-03-25T18:19:03.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.741+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6cb4001b10 con 0x7f6cbc1430c0 2024-03-25T18:19:03.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.741+0000 7f6cb2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:03.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.741+0000 7f6cb2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:03.742 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.741+0000 7f6cb2ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:03.743 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:19:03.743 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:03.743 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:03.743 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:03.743 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:03.743 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:03.743 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:19:03.743 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:19:03.743 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:19:03.743 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:03.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.1 v1:172.21.15.117:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 177667455 0 0) 0x7f6cbc13be70 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 >> v1:172.21.15.37:6790/0 conn(0x7f6cbc1430c0 legacy=0x7f6cbc1e2fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:03.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 >> v1:172.21.15.37:6789/0 conn(0x7f6cbc13f710 legacy=0x7f6cbc1d76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:03.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cb2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:03.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cb2ffd640 1 monclient: found mon.b 2024-03-25T18:19:03.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cb2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:03.757 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cb2ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc131980 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cb2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cb2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cb2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cc46f1640 5 monclient: authenticate success, global_id 4729 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cc46f1640 10 monclient: _renew_subs 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cc46f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cc46f1640 1 -- 172.21.15.37:0/3333320801 --> v1:172.21.15.117:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cbc135c60 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cc46f1640 10 client.4729.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cc46f1640 10 monclient: _renew_subs 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cc46f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cc46f1640 1 -- 172.21.15.37:0/3333320801 --> v1:172.21.15.117:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cbc0b7770 con 0x7f6cbc0a98f0 2024-03-25T18:19:03.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.755+0000 7f6cc46f1640 1 librados: init done 2024-03-25T18:19:03.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.756+0000 7f6cc46f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:03.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.756+0000 7f6cc46f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:03.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.757+0000 7f6cc46f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:19:03.759 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:03.757+0000 7f6cc46f1640 1 -- 172.21.15.37:0/3333320801 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cbc1371d0 con 0x7f6cbc0a98f0 2024-03-25T18:19:04.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.052+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.1 v1:172.21.15.117:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6cb80011f0 con 0x7f6cbc0a98f0 2024-03-25T18:19:04.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.052+0000 7f6cb2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:04.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.052+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.1 v1:172.21.15.117:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f6cb8004cb0 con 0x7f6cbc0a98f0 2024-03-25T18:19:04.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.052+0000 7f6cb2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:04.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.052+0000 7f6cb2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:04.053 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.052+0000 7f6cb2ffd640 10 monclient: dump: 2024-03-25T18:19:04.053 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:19:04.054 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:04.054 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:04.054 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:04.054 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:04.054 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:04.054 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:19:04.054 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:19:04.054 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:19:04.054 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:04.054 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.052+0000 7f6cc0c8a640 4 set_mon_vals no callback set 2024-03-25T18:19:04.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.053+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.1 v1:172.21.15.117:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f6cb801acc0 con 0x7f6cbc0a98f0 2024-03-25T18:19:04.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.053+0000 7f6cc1c8c640 10 monclient: get_auth_request con 0x7f6ca006df10 auth_method 0 2024-03-25T18:19:04.055 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.053+0000 7f6cb2ffd640 10 client.4729.objecter ms_handle_connect 0x7f6ca006df10 2024-03-25T18:19:04.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.058+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.1 v1:172.21.15.117:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7f6cb80549c0 con 0x7f6cbc0a98f0 2024-03-25T18:19:04.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.058+0000 7f6cb2ffd640 10 client.4729.objecter ms_dispatch 0x7f6cbc000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:04.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.058+0000 7f6cb2ffd640 3 client.4729.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:04.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.058+0000 7f6cb2ffd640 3 client.4729.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:04.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.059+0000 7f6cb2ffd640 20 client.4729.objecter dump_active .. 0 homeless 2024-03-25T18:19:04.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.059+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.1 v1:172.21.15.117:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6cb8050300 con 0x7f6cbc0a98f0 2024-03-25T18:19:04.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.059+0000 7f6cb2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:04.060 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.059+0000 7f6cb2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:04.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.287+0000 7f6cc46f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:04.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.287+0000 7f6cc46f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:04.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.287+0000 7f6cc46f1640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.117:6789/0 2024-03-25T18:19:04.289 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.287+0000 7f6cc46f1640 1 -- 172.21.15.37:0/3333320801 --> v1:172.21.15.117:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6cbc074cc0 con 0x7f6cbc0a98f0 2024-03-25T18:19:04.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.313+0000 7f6cb2ffd640 1 -- 172.21.15.37:0/3333320801 <== mon.1 v1:172.21.15.117:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7f6cb8052400 con 0x7f6cbc0a98f0 2024-03-25T18:19:04.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.313+0000 7f6cb2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:04.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.313+0000 7f6cb2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:04.315 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:19:04.315 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:04.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 10 librados: watch_flush enter 2024-03-25T18:19:04.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 10 librados: watch_flush exit 2024-03-25T18:19:04.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 20 client.4729.objecter shutdown clearing up homeless session... 2024-03-25T18:19:04.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 10 client.4729.objecter successfully canceled tick 2024-03-25T18:19:04.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 1 -- 172.21.15.37:0/3333320801 >> v1:172.21.15.117:6800/3703993921 conn(0x7f6ca006df10 legacy=0x7f6ca00703d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:04.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 10 monclient: shutdown 2024-03-25T18:19:04.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:04.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 1 -- 172.21.15.37:0/3333320801 >> v1:172.21.15.117:6789/0 conn(0x7f6cbc0a98f0 legacy=0x7f6cbc130f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:04.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 1 -- 172.21.15.37:0/3333320801 shutdown_connections 2024-03-25T18:19:04.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 1 -- 172.21.15.37:0/3333320801 >> 172.21.15.37:0/3333320801 conn(0x7f6cbc0b7ea0 msgr2=0x7f6cbc074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:04.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 1 -- 172.21.15.37:0/3333320801 shutdown_connections 2024-03-25T18:19:04.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 1 -- 172.21.15.37:0/3333320801 wait complete. 2024-03-25T18:19:04.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:04.316+0000 7f6cc46f1640 1 librados: shutdown 2024-03-25T18:19:04.329 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:11.135 DEBUG:teuthology.orchestra.run.smithi037:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:11.254 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.253+0000 7f0141ce8640 10 monclient: get_monmap_and_config 2024-03-25T18:19:11.254 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.253+0000 7f0141ce8640 10 monclient: build_initial_monmap 2024-03-25T18:19:11.254 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.253+0000 7f0141ce8640 10 monclient: monmap: 2024-03-25T18:19:11.254 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:19:11.255 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:11.255 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:19:11.254523+0000 2024-03-25T18:19:11.255 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:19:11.254523+0000 2024-03-25T18:19:11.255 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:11.255 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:11.255 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:19:11.255 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-c 2024-03-25T18:19:11.255 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-b 2024-03-25T18:19:11.255 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:11.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.254+0000 7f0141ce8640 1 Processor -- start 2024-03-25T18:19:11.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.254+0000 7f0141ce8640 1 -- start start 2024-03-25T18:19:11.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.254+0000 7f0141ce8640 10 monclient: init 2024-03-25T18:19:11.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.254+0000 7f0141ce8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:11.256 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.255+0000 7f0141ce8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:11.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.255+0000 7f0141ce8640 10 monclient(hunting): picked mon.noname-b con 0x7f013c0b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:19:11.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.255+0000 7f0141ce8640 10 monclient(hunting): picked mon.noname-c con 0x7f013c0b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:19:11.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.255+0000 7f0141ce8640 10 monclient(hunting): picked mon.noname-a con 0x7f013c14be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:19:11.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.255+0000 7f0141ce8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013c0af940 con 0x7f013c14be90 2024-03-25T18:19:11.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.255+0000 7f0141ce8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013c0b0a70 con 0x7f013c0b1730 2024-03-25T18:19:11.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.255+0000 7f0141ce8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013c0b0d50 con 0x7f013c0b4d60 2024-03-25T18:19:11.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.255+0000 7f0141ce8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:11.257 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.255+0000 7f0141ce8640 10 monclient(hunting): authenticate will time out at 7506.507503s 2024-03-25T18:19:11.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.263+0000 7f013bfff640 1 --1- >> v1:172.21.15.37:6789/0 conn(0x7f013c14be90 0x7f013c14e280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.37:6789/0 says I am v1:172.21.15.37:43176/0 (socket says 172.21.15.37:43176) 2024-03-25T18:19:11.264 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.263+0000 7f013bfff640 1 -- 172.21.15.37:0/4268119366 learned_addr learned my addr 172.21.15.37:0/4268119366 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:19:11.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.276+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 122428881 0 0) 0x7f013c0af940 con 0x7f013c14be90 2024-03-25T18:19:11.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.276+0000 7f013a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:11.277 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.276+0000 7f013a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:11.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.276+0000 7f013a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:11.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.276+0000 7f013a7fc640 10 monclient(hunting): my global_id is 4746 2024-03-25T18:19:11.278 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.276+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0128003520 con 0x7f013c14be90 2024-03-25T18:19:11.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.290+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4135558626 0 0) 0x7f013c0b0a70 con 0x7f013c0b1730 2024-03-25T18:19:11.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.290+0000 7f013a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:11.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.290+0000 7f013a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:11.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.290+0000 7f013a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:11.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.290+0000 7f013a7fc640 10 monclient(hunting): my global_id is 4732 2024-03-25T18:19:11.291 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.290+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f013c0af940 con 0x7f013c0b1730 2024-03-25T18:19:11.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.291+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 305686148 0 0) 0x7f013c0b0d50 con 0x7f013c0b4d60 2024-03-25T18:19:11.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.291+0000 7f013a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:11.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.291+0000 7f013a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:11.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.291+0000 7f013a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:11.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.291+0000 7f013a7fc640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:19:11.292 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.291+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f013c0b0a70 con 0x7f013c0b4d60 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2480167377 0 0) 0x7f0128003520 con 0x7f013c14be90 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 >> v1:172.21.15.37:6790/0 conn(0x7f013c0b4d60 legacy=0x7f013c13f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 >> v1:172.21.15.117:6789/0 conn(0x7f013c0b1730 legacy=0x7f013c0b1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.37:6789/0 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 --> v1:172.21.15.37:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013c0b1030 con 0x7f013c14be90 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0134001a10 con 0x7f013c14be90 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f0141ce8640 5 monclient: authenticate success, global_id 4746 2024-03-25T18:19:11.296 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f0141ce8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f013a7fc640 10 monclient: dump: 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:11.297 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.294+0000 7f0141ce8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:11.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.312+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 <== mon.0 v1:172.21.15.37:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0134001c30 con 0x7f013c14be90 2024-03-25T18:19:11.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.312+0000 7f013a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:11.313 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.312+0000 7f013a7fc640 1 -- 172.21.15.37:0/4268119366 <== mon.0 v1:172.21.15.37:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0134001fc0 con 0x7f013c14be90 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.312+0000 7f0141ce8640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.312+0000 7f0141ce8640 4 set_mon_vals no callback set 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.312+0000 7f013a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.312+0000 7f013a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.312+0000 7f013a7fc640 10 monclient: dump: 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:19:11.314 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.313+0000 7f0141ce8640 10 monclient: shutdown 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.313+0000 7f0141ce8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.313+0000 7f0141ce8640 1 -- 172.21.15.37:0/4268119366 >> v1:172.21.15.37:6789/0 conn(0x7f013c14be90 legacy=0x7f013c14e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.313+0000 7f0141ce8640 1 -- 172.21.15.37:0/4268119366 shutdown_connections 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.313+0000 7f0141ce8640 1 -- 172.21.15.37:0/4268119366 >> 172.21.15.37:0/4268119366 conn(0x7f013c0617a0 msgr2=0x7f013c061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.313+0000 7f0141ce8640 1 -- 172.21.15.37:0/4268119366 shutdown_connections 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.313+0000 7f0141ce8640 1 -- 172.21.15.37:0/4268119366 wait complete. 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.314+0000 7f0141ce8640 10 monclient: build_initial_monmap 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.314+0000 7f0141ce8640 10 monclient: monmap: 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:epoch 0 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:last_changed 0.000000 2024-03-25T18:19:11.316 INFO:teuthology.misc.health.smithi037.stderr:created 0.000000 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 1 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.noname-a 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.noname-b 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.noname-c 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.314+0000 7f0141ce8640 1 librados: starting msgr at 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.314+0000 7f0141ce8640 1 librados: starting objecter 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.314+0000 7f0141ce8640 1 Processor -- start 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 1 -- start start 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 1 librados: setting wanted keys 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 1 librados: calling monclient init 2024-03-25T18:19:11.317 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 10 monclient: init 2024-03-25T18:19:11.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:11.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:11.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 10 monclient(hunting): picked mon.noname-b con 0x7f013c0b1730 addr v1:172.21.15.117:6789/0 2024-03-25T18:19:11.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 10 monclient(hunting): picked mon.noname-c con 0x7f013c0b4d60 addr v1:172.21.15.37:6790/0 2024-03-25T18:19:11.318 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 10 monclient(hunting): picked mon.noname-a con 0x7f013c14be90 addr v1:172.21.15.37:6789/0 2024-03-25T18:19:11.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 1 -- --> v1:172.21.15.37:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013c141da0 con 0x7f013c14be90 2024-03-25T18:19:11.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013c1409a0 con 0x7f013c0b1730 2024-03-25T18:19:11.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 1 -- --> v1:172.21.15.37:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f013c1456f0 con 0x7f013c0b4d60 2024-03-25T18:19:11.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:11.319 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.315+0000 7f0141ce8640 10 monclient(hunting): authenticate will time out at 7506.567965s 2024-03-25T18:19:11.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.333+0000 7f013affd640 1 --1- >> v1:172.21.15.37:6790/0 conn(0x7f013c0b4d60 0x7f013c142b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.37:6790/0 says I am v1:172.21.15.37:36594/0 (socket says 172.21.15.37:36594) 2024-03-25T18:19:11.334 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.333+0000 7f013affd640 1 -- 172.21.15.37:0/3951001651 learned_addr learned my addr 172.21.15.37:0/3951001651 (peer_addr_for_me v1:172.21.15.37:0/0) 2024-03-25T18:19:11.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.338+0000 7f0133fff640 10 client.?.objecter ms_handle_connect 0x7f013c0b4d60 2024-03-25T18:19:11.339 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.338+0000 7f0133fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:11.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.340+0000 7f0133fff640 10 client.?.objecter ms_handle_connect 0x7f013c0b1730 2024-03-25T18:19:11.341 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.340+0000 7f0133fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:11.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.350+0000 7f0133fff640 10 client.?.objecter ms_handle_connect 0x7f013c14be90 2024-03-25T18:19:11.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.350+0000 7f0133fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:11.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.350+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.2 v1:172.21.15.37:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193841555 0 0) 0x7f013c1456f0 con 0x7f013c0b4d60 2024-03-25T18:19:11.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.351+0000 7f0133fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:11.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.351+0000 7f0133fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:11.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.351+0000 7f0133fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:11.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.351+0000 7f0133fff640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:19:11.352 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.351+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 --> v1:172.21.15.37:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f011c004eb0 con 0x7f013c0b4d60 2024-03-25T18:19:11.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.352+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740424452 0 0) 0x7f013c1409a0 con 0x7f013c0b1730 2024-03-25T18:19:11.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.352+0000 7f0133fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:11.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.352+0000 7f0133fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:11.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.352+0000 7f0133fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:11.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.352+0000 7f0133fff640 10 monclient(hunting): my global_id is 4735 2024-03-25T18:19:11.354 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.352+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f013c1456f0 con 0x7f013c0b1730 2024-03-25T18:19:11.368 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.367+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.0 v1:172.21.15.37:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1549870139 0 0) 0x7f013c141da0 con 0x7f013c14be90 2024-03-25T18:19:11.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.367+0000 7f0133fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:11.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.367+0000 7f0133fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:11.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.367+0000 7f0133fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:11.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.367+0000 7f0133fff640 10 monclient(hunting): my global_id is 4749 2024-03-25T18:19:11.369 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.367+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 --> v1:172.21.15.37:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f013c1409a0 con 0x7f013c14be90 2024-03-25T18:19:11.374 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.373+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.2 v1:172.21.15.37:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1964565349 0 0) 0x7f011c004eb0 con 0x7f013c0b4d60 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.373+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 --> v1:172.21.15.37:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f013c141da0 con 0x7f013c0b4d60 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.373+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.2 v1:172.21.15.37:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f012c001a30 con 0x7f013c0b4d60 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.373+0000 7f0133fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.373+0000 7f0133fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.373+0000 7f0133fff640 10 monclient(hunting): dump: 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:19:11.375 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:19:11.376 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:11.376 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.373+0000 7f0133fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.383+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 854248356 0 0) 0x7f013c1456f0 con 0x7f013c0b1730 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.384+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 --> v1:172.21.15.117:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f011c004eb0 con 0x7f013c0b1730 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.384+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0124001870 con 0x7f013c0b1730 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.384+0000 7f0133fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.384+0000 7f0133fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.384+0000 7f0133fff640 10 monclient(hunting): dump: 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:11.385 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:11.386 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:19:11.386 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:19:11.386 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:19:11.386 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:11.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.392+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.0 v1:172.21.15.37:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2098124905 0 0) 0x7f013c1409a0 con 0x7f013c14be90 2024-03-25T18:19:11.393 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.392+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 --> v1:172.21.15.37:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f013c1456f0 con 0x7f013c14be90 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.392+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.0 v1:172.21.15.37:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f0134001da0 con 0x7f013c14be90 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.392+0000 7f0133fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.392+0000 7f0133fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.392+0000 7f0133fff640 10 monclient(hunting): dump: 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:19:11.394 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:11.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.405+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.2 v1:172.21.15.37:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1571510666 0 0) 0x7f013c141da0 con 0x7f013c0b4d60 2024-03-25T18:19:11.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 >> v1:172.21.15.117:6789/0 conn(0x7f013c0b1730 legacy=0x7f013c142410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:11.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 >> v1:172.21.15.37:6789/0 conn(0x7f013c14be90 legacy=0x7f013c1e24a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:11.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0133fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:11.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0133fff640 1 monclient: found mon.c 2024-03-25T18:19:11.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0133fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:11.407 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0133fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 --> v1:172.21.15.37:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f013c145880 con 0x7f013c0b4d60 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0133fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0133fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0133fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0141ce8640 5 monclient: authenticate success, global_id 4736 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0141ce8640 10 monclient: _renew_subs 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0141ce8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0141ce8640 1 -- 172.21.15.37:0/3951001651 --> v1:172.21.15.37:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f013c137130 con 0x7f013c0b4d60 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0141ce8640 10 client.4736.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0141ce8640 10 monclient: _renew_subs 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0141ce8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0141ce8640 1 -- 172.21.15.37:0/3951001651 --> v1:172.21.15.37:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f013c140380 con 0x7f013c0b4d60 2024-03-25T18:19:11.408 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.406+0000 7f0141ce8640 1 librados: init done 2024-03-25T18:19:11.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.407+0000 7f0141ce8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:11.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.407+0000 7f0141ce8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:11.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.407+0000 7f0141ce8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:19:11.409 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.407+0000 7f0141ce8640 1 -- 172.21.15.37:0/3951001651 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f013c1401f0 con 0x7f013c0b4d60 2024-03-25T18:19:11.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.750+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.2 v1:172.21.15.37:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f012c001c70 con 0x7f013c0b4d60 2024-03-25T18:19:11.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.750+0000 7f0133fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:11.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.750+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.2 v1:172.21.15.37:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 703537395 0 0) 0x7f012c003a60 con 0x7f013c0b4d60 2024-03-25T18:19:11.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.750+0000 7f0133fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:11.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.750+0000 7f0133fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:11.752 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.750+0000 7f0133fff640 10 monclient: dump: 2024-03-25T18:19:11.752 INFO:teuthology.misc.health.smithi037.stderr:epoch 1 2024-03-25T18:19:11.752 INFO:teuthology.misc.health.smithi037.stderr:fsid f42897d2-ff81-4b7f-bd91-55865e0be8e5 2024-03-25T18:19:11.752 INFO:teuthology.misc.health.smithi037.stderr:last_changed 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.752 INFO:teuthology.misc.health.smithi037.stderr:created 2024-03-25T18:06:51.758639+0000 2024-03-25T18:19:11.752 INFO:teuthology.misc.health.smithi037.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:11.753 INFO:teuthology.misc.health.smithi037.stderr:election_strategy: 3 2024-03-25T18:19:11.753 INFO:teuthology.misc.health.smithi037.stderr:0: v1:172.21.15.37:6789/0 mon.a 2024-03-25T18:19:11.753 INFO:teuthology.misc.health.smithi037.stderr:1: v1:172.21.15.117:6789/0 mon.b 2024-03-25T18:19:11.753 INFO:teuthology.misc.health.smithi037.stderr:2: v1:172.21.15.37:6790/0 mon.c 2024-03-25T18:19:11.753 INFO:teuthology.misc.health.smithi037.stderr: 2024-03-25T18:19:11.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.750+0000 7f0139ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:11.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.751+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.2 v1:172.21.15.37:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 1525401134 0 0) 0x7f012c003c40 con 0x7f013c0b4d60 2024-03-25T18:19:11.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.752+0000 7f013b7fe640 10 monclient: get_auth_request con 0x7f011c06e0f0 auth_method 0 2024-03-25T18:19:11.753 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.752+0000 7f0133fff640 10 client.4736.objecter ms_handle_connect 0x7f011c06e0f0 2024-03-25T18:19:11.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.756+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.2 v1:172.21.15.37:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 3521377766 0 0) 0x7f012c054850 con 0x7f013c0b4d60 2024-03-25T18:19:11.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.756+0000 7f0133fff640 10 client.4736.objecter ms_dispatch 0x7f013c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:19:11.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.756+0000 7f0133fff640 3 client.4736.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:19:11.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.756+0000 7f0133fff640 3 client.4736.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:19:11.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.756+0000 7f0133fff640 20 client.4736.objecter dump_active .. 0 homeless 2024-03-25T18:19:11.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.756+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.2 v1:172.21.15.37:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f012c050360 con 0x7f013c0b4d60 2024-03-25T18:19:11.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.756+0000 7f0133fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:11.758 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.756+0000 7f0133fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:11.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.988+0000 7f0141ce8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:11.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.988+0000 7f0141ce8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:11.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.988+0000 7f0141ce8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.37:6790/0 2024-03-25T18:19:11.989 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:11.988+0000 7f0141ce8640 1 -- 172.21.15.37:0/3951001651 --> v1:172.21.15.37:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f013c140b30 con 0x7f013c0b4d60 2024-03-25T18:19:12.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.020+0000 7f0133fff640 1 -- 172.21.15.37:0/3951001651 <== mon.2 v1:172.21.15.37:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3782177420) 0x7f012c052290 con 0x7f013c0b4d60 2024-03-25T18:19:12.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.020+0000 7f0133fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:12.021 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.020+0000 7f0133fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:12.021 INFO:teuthology.misc.health.smithi037.stdout: 2024-03-25T18:19:12.022 INFO:teuthology.misc.health.smithi037.stdout:{"epoch":11,"fsid":"f42897d2-ff81-4b7f-bd91-55865e0be8e5","created":"2024-03-25T18:07:42.095418+0000","modified":"2024-03-25T18:17:46.023267+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:17:46.023267+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9b8b387b-0c1c-40a3-9b30-bbde42ac181e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"abc7c058-668d-4fef-90fa-3d0d17a0aa34","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"ddddbd2b-0e5f-4930-911b-9bac20d19eeb","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"e00e8f64-0c01-4f09-a358-ecc0a60e53f4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"9fc4103b-00f7-42b5-b5b6-b00d69c9bd4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"028242bd-e284-45f2-bd64-e250bcaea3f9","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"b9f96f7d-77ce-40be-a2eb-041c13198fbf","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"57e48c8b-9e7c-4113-9d08-0c858b9faf5b","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:12.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.022+0000 7f0141ce8640 10 librados: watch_flush enter 2024-03-25T18:19:12.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.022+0000 7f0141ce8640 10 librados: watch_flush exit 2024-03-25T18:19:12.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.022+0000 7f0141ce8640 20 client.4736.objecter shutdown clearing up homeless session... 2024-03-25T18:19:12.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.022+0000 7f0141ce8640 10 client.4736.objecter successfully canceled tick 2024-03-25T18:19:12.024 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.022+0000 7f0141ce8640 1 -- 172.21.15.37:0/3951001651 >> v1:172.21.15.117:6800/3703993921 conn(0x7f011c06e0f0 legacy=0x7f011c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:12.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.022+0000 7f0141ce8640 10 monclient: shutdown 2024-03-25T18:19:12.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.022+0000 7f0141ce8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:12.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.022+0000 7f0141ce8640 1 -- 172.21.15.37:0/3951001651 >> v1:172.21.15.37:6790/0 conn(0x7f013c0b4d60 legacy=0x7f013c142b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:12.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.023+0000 7f0141ce8640 1 -- 172.21.15.37:0/3951001651 shutdown_connections 2024-03-25T18:19:12.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.023+0000 7f0141ce8640 1 -- 172.21.15.37:0/3951001651 >> 172.21.15.37:0/3951001651 conn(0x7f013c0617a0 msgr2=0x7f013c13c600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:12.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.023+0000 7f0141ce8640 1 -- 172.21.15.37:0/3951001651 shutdown_connections 2024-03-25T18:19:12.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.023+0000 7f0141ce8640 1 -- 172.21.15.37:0/3951001651 wait complete. 2024-03-25T18:19:12.025 INFO:teuthology.misc.health.smithi037.stderr:2024-03-25T18:19:12.023+0000 7f0141ce8640 1 librados: shutdown 2024-03-25T18:19:12.036 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:12.036 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:19:12.038 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:19:12.038 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:19:12.038 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:19:13.978 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:11:26.340 35479== 2024-03-25T18:19:13.978 INFO:tasks.ceph.osd.0.smithi037.stderr:==00:00:11:26.340 35479== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:19:14.016 INFO:tasks.ceph.osd.0.smithi037.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:19:16.411 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:51.756 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:16.457 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:51.770 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:16.460 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:51.773 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:16.468 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:51.813 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:16.496 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:11:51.808 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:16.503 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:11:51.849 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:16.512 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:51.857 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:16.523 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:51.868 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:16.552 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:11:51.897 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:18.140 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:19:18.140 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:19:18.141 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:19:18.141 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:19:18.141 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:19:20.042 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:11:32.420 35476== 2024-03-25T18:19:20.043 INFO:tasks.ceph.osd.1.smithi037.stderr:==00:00:11:32.420 35476== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:19:20.088 INFO:tasks.ceph.osd.1.smithi037.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:19:24.243 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:19:24.243 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:19:24.243 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:19:24.244 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:19:24.244 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:19:26.081 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:11:38.446 35478== 2024-03-25T18:19:26.081 INFO:tasks.ceph.osd.2.smithi037.stderr:==00:00:11:38.446 35478== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:19:26.121 INFO:tasks.ceph.osd.2.smithi037.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:19:30.346 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:19:30.346 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:19:30.347 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:19:30.347 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:19:30.347 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:19:32.114 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:11:44.487 35477== 2024-03-25T18:19:32.114 INFO:tasks.ceph.osd.3.smithi037.stderr:==00:00:11:44.487 35477== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:19:32.151 INFO:tasks.ceph.osd.3.smithi037.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:19:36.448 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:19:36.448 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:19:36.449 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:19:36.449 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:19:36.449 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:19:36.488 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:12:11.833 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:36.491 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:12:11.836 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:36.509 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:12:11.855 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:36.540 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:12:11.886 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:36.544 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:12:11.890 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:36.555 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:12:11.900 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:36.559 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:12:11.872 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:36.563 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:12:11.876 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:36.588 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:12:11.901 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:38.359 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:11:50.739 35533== 2024-03-25T18:19:38.359 INFO:tasks.ceph.osd.4.smithi117.stderr:==00:00:11:50.739 35533== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:19:38.395 INFO:tasks.ceph.osd.4.smithi117.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:19:42.552 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:19:42.552 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi117 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:19:42.552 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:19:42.553 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:19:42.553 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:19:44.396 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:11:56.751 35535== 2024-03-25T18:19:44.396 INFO:tasks.ceph.osd.5.smithi117.stderr:==00:00:11:56.751 35535== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:19:44.435 INFO:tasks.ceph.osd.5.smithi117.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:19:48.655 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:19:48.655 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi117 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:19:48.655 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:19:48.656 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:19:48.656 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:19:50.528 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:12:02.901 35534== 2024-03-25T18:19:50.528 INFO:tasks.ceph.osd.6.smithi117.stderr:==00:00:12:02.901 35534== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:19:50.573 INFO:tasks.ceph.osd.6.smithi117.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:19:54.757 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:19:54.757 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi117 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:19:54.758 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:19:54.758 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:19:54.758 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:19:56.201 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:12:08.556 35536== 2024-03-25T18:19:56.202 INFO:tasks.ceph.osd.7.smithi117.stderr:==00:00:12:08.556 35536== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:19:56.231 INFO:tasks.ceph.osd.7.smithi117.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:19:56.535 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:12:31.880 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:56.536 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:12:31.882 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:56.548 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:12:31.893 35164== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:56.577 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:12:31.890 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:56.579 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:12:31.892 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:56.584 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:12:31.929 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:56.585 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:12:31.930 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:56.589 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:12:31.902 34989== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:56.596 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:12:31.941 35166== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:00.860 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:00.860 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi117 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:20:00.860 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:20:00.860 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:20:00.861 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:20:00.861 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:00.896 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:20:00.896 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:20:00.897 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:00.898 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:20:00.897+0000 7f402ee90640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i x (PID: 34987) UID: 0 2024-03-25T18:20:00.898 INFO:tasks.ceph.mgr.x.smithi117.stderr:2024-03-25T18:20:00.897+0000 7f402ee90640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:20:00.922 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:20:00.922 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:20:00.922 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:20:00.922 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:00.953 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-03-25T18:20:00.940+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35131) UID: 0 2024-03-25T18:20:00.954 INFO:tasks.ceph.mon.a.smithi037.stderr:2024-03-25T18:20:00.942+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:20:01.541 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:12:36.886 35164== 2024-03-25T18:20:01.541 INFO:tasks.ceph.mon.a.smithi037.stderr:==00:00:12:36.886 35164== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:01.550 INFO:tasks.ceph.mon.a.smithi037.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:07.025 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:07.025 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:20:07.025 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:20:07.026 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:20:07.026 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:07.042 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-03-25T18:20:07.029+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35133) UID: 0 2024-03-25T18:20:07.042 INFO:tasks.ceph.mon.c.smithi037.stderr:2024-03-25T18:20:07.031+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:20:07.683 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:12:43.028 35166== 2024-03-25T18:20:07.683 INFO:tasks.ceph.mon.c.smithi037.stderr:==00:00:12:43.028 35166== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:07.692 INFO:tasks.ceph.mon.c.smithi037.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:13.127 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:13.128 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi037 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:20:13.128 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:20:13.129 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:20:13.129 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:20:13.145 INFO:tasks.ceph.mon.b.smithi117.stderr:2024-03-25T18:20:13.132+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 34985) UID: 0 2024-03-25T18:20:13.145 INFO:tasks.ceph.mon.b.smithi117.stderr:2024-03-25T18:20:13.134+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:20:20.738 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:12:56.050 34989== 2024-03-25T18:20:20.738 INFO:tasks.ceph.mon.b.smithi117.stderr:==00:00:12:56.050 34989== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:20:20.747 INFO:tasks.ceph.mon.b.smithi117.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:20:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:20:25.332 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi117 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:20:25.332 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:20:25.332 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:20:25.333 DEBUG:teuthology.orchestra.run.smithi037:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:20:25.380 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-25T18:20:25.380 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:20:25.499 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-25T18:20:25.500 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:20:25.565 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-25T18:20:25.565 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:20:25.623 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi037.front.sepia.ceph.com 2024-03-25T18:20:25.623 DEBUG:teuthology.orchestra.run.smithi037:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:20:25.809 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-25T18:20:25.809 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:20:25.933 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-25T18:20:25.933 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:20:25.989 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-25T18:20:25.989 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:20:26.047 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi117.front.sepia.ceph.com 2024-03-25T18:20:26.047 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:20:26.105 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:20:26.108 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265/data/mon.a.tgz 2024-03-25T18:20:26.108 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-03-25T18:20:26.125 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.ivX1Nizxio 2024-03-25T18:20:26.126 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ivX1Nizxio 2024-03-25T18:20:26.233 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.ivX1Nizxio 2024-03-25T18:20:26.322 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.ivX1Nizxio is 134KB 2024-03-25T18:20:26.346 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.ivX1Nizxio 2024-03-25T18:20:26.361 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265/data/mon.c.tgz 2024-03-25T18:20:26.362 DEBUG:teuthology.orchestra.run.smithi037:> mktemp 2024-03-25T18:20:26.417 INFO:teuthology.orchestra.run.smithi037.stdout:/tmp/tmp.t1a2YTYkdb 2024-03-25T18:20:26.417 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.t1a2YTYkdb 2024-03-25T18:20:26.508 DEBUG:teuthology.orchestra.run.smithi037:> sudo chmod 0666 /tmp/tmp.t1a2YTYkdb 2024-03-25T18:20:26.593 DEBUG:teuthology.orchestra.remote:smithi037:/tmp/tmp.t1a2YTYkdb is 134KB 2024-03-25T18:20:26.645 DEBUG:teuthology.orchestra.run.smithi037:> rm -fr /tmp/tmp.t1a2YTYkdb 2024-03-25T18:20:26.659 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265/data/mon.b.tgz 2024-03-25T18:20:26.660 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-03-25T18:20:26.675 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.EeOWJSXZpu 2024-03-25T18:20:26.675 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.EeOWJSXZpu 2024-03-25T18:20:26.772 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.EeOWJSXZpu 2024-03-25T18:20:26.855 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.EeOWJSXZpu is 134KB 2024-03-25T18:20:26.876 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.EeOWJSXZpu 2024-03-25T18:20:26.890 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:20:26.890 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:20:26.892 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:20:27.164 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:20:27.341 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:20:27.343 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265/remote/smithi037/crash 2024-03-25T18:20:27.344 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:20:27.371 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265/remote/smithi117/crash 2024-03-25T18:20:27.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:20:27.399 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:20:27.399 DEBUG:teuthology.orchestra.run.smithi037:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:20:27.415 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:20:27.439 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39705.log 2024-03-25T18:20:27.439 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:20:27.439 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39705.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41813.log 2024-03-25T18:20:27.481 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39427.log 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39705.log.gz 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41813.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41461.log 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39427.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40914.log 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41813.log.gz 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39153.log 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39427.log.gz 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40914.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40797.log 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41461.log.gz 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39153.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39310.log 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40914.log.gz 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39153.log.gz 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38780.log 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40797.log: /var/log/ceph/ceph-client.admin.39310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41382.log 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38780.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40797.log.gz 2024-03-25T18:20:27.482 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39861.log 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39310.log.gz 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41382.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38780.log.gz 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36510.log 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41891.log 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39861.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41382.log.gz 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.36510.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39627.log 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39861.log.gz 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41891.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.36510.log.gz 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41891.log.gz 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39627.log: /var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39627.log.gz 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/valgrind/osd.2.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:20:27.483 INFO:teuthology.orchestra.run.smithi037.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/valgrind/osd.0.log: /var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41031.log 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/valgrind/mon.a.log: /var/log/ceph/valgrind/osd.3.log: 69.6% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39231.log 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr: 78.5% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41500.log 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39548.log 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41031.log.gz 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41500.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40992.log 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39231.log.gz 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39548.log: gzip -5 --verbose 86.6% -- /var/log/ceph/ceph-client.admin.40407.log 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41500.log.gz 2024-03-25T18:20:27.484 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39034.log 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39548.log.gz 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39900.log 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40992.log.gz 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39034.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41148.log 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40407.log.gz 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40563.log 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39034.log.gz 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41148.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39744.log 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39900.log.gz 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41617.log 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41148.log.gz 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39744.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41774.log 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40563.log.gz 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41617.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39744.log.gz 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42203.log 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41774.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40719.log 2024-03-25T18:20:27.485 INFO:teuthology.orchestra.run.smithi037.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41617.log.gz 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.42203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40758.log 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41774.log.gz 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40641.log 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.42203.log.gz 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40719.log: /var/log/ceph/ceph-client.admin.40758.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41539.log 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40719.log.gz 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39509.log 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40758.log.gz 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41539.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40641.log.gz 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40602.log 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41539.log.gz 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39509.log: /var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37545.log 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39509.log.gz 2024-03-25T18:20:27.486 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40602.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41735.log 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40602.log.gz 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr:gzip 93.5% -5 --verbose -- replaced with /var/log/ceph/ceph.audit.log.gz -- 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr: /var/log/ceph/ceph-client.admin.42008.log 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.37545.log: /var/log/ceph/ceph-client.admin.41735.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38995.log 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.42008.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41304.log 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.37545.log.gz 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41735.log.gz 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.38995.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41265.log 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42008.log.gz 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39114.log 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38995.log.gz 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41578.log 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41304.log.gz 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39114.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41265.log.gz 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41070.log 2024-03-25T18:20:27.487 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41578.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39114.log.gz 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40173.log 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41578.log.gz 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40524.log 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41070.log.gz 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40173.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41422.log 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41695.log 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40173.log.gz 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40524.log.gz 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41422.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35475.log 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41695.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39388.log 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr: 86.1%/var/log/ceph/ceph-client.admin.35475.log: -- replaced with /var/log/ceph/ceph-client.admin.41422.log.gz 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41969.log 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41695.log.gz 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40875.log 2024-03-25T18:20:27.488 INFO:teuthology.orchestra.run.smithi037.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35475.log.gz 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41969.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41656.log 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39388.log.gz 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41226.log 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41969.log.gz 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41656.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39271.log 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40875.log.gz 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40251.log 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41656.log.gz 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39271.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35502.log 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.41226.log.gz 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39822.log 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39271.log.gz 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35502.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40251.log.gz 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39783.log 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39822.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.35502.log.gz 2024-03-25T18:20:27.489 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39783.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39978.log 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39822.log.gz 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.3.log: gzip 86.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.39783.log.gz 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph-client.admin.40056.log 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40446.log 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40056.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39192.log 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39978.log.gz 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40017.log 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40056.log.gz 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39192.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39939.log 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40446.log.gz 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40017.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40368.log 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39192.log.gz 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39939.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40134.log 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40017.log.gz 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42281.log 2024-03-25T18:20:27.490 INFO:teuthology.orchestra.run.smithi037.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39939.log.gz 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40134.log: gzip -5 --verbose 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40368.log.gz 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr: -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.42281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39349.log 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40134.log.gz 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40680.log 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.42281.log.gz 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41187.log 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39349.log: /var/log/ceph/ceph-client.admin.40680.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41852.log 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41187.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39349.log.gz 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose 87.3% -- /var/log/ceph/ceph-client.admin.41343.log -- replaced with /var/log/ceph/ceph-client.admin.40680.log.gz 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr:gzip 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41187.log.gz 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr: -5/var/log/ceph/ceph-client.admin.41852.log: --verbose -- /var/log/ceph/ceph-client.admin.42047.log 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39666.log 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr: 86.6%/var/log/ceph/ceph-client.admin.42047.log: -- replaced with /var/log/ceph/ceph-client.admin.41852.log.gz 2024-03-25T18:20:27.491 INFO:teuthology.orchestra.run.smithi037.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41343.log.gzgzip 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr: -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:gzip 86.9% -5 --verbose -- /var/log/ceph/ceph-client.admin.41109.log 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39666.log: -- replaced with /var/log/ceph/ceph-client.admin.42047.log.gz 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29321.log 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39666.log.gz 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35649.log 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41109.log: /var/log/ceph/ceph.tmp-client.admin.29321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42164.log 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29321.log.gz 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35649.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41109.log.gz 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35649.log.gz 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40212.log 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.42164.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40212.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42164.log.gz 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40212.log.gz 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40836.log 2024-03-25T18:20:27.492 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40329.log 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr: 92.9%/var/log/ceph/ceph-osd.2.log: -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40836.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40836.log.gz 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42242.log 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39587.log 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.42242.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40329.log.gz 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42242.log.gz 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42086.log 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39587.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41930.log 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.42086.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39587.log.gz 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42086.log.gz 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr: 93.8%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39467.log 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr: -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.41930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40485.log 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39467.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40290.log 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.41930.log.gz 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40485.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39467.log.gz 2024-03-25T18:20:27.493 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40485.log.gz 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr: --verbose -- /var/log/ceph/ceph-client.admin.42125.log 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40290.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35173.log 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr: 86.0%/var/log/ceph/ceph-client.admin.42125.log: -- replaced with /var/log/ceph/ceph-client.admin.40290.log.gz 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40095.log 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.42125.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39075.log 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.35173.log: /var/log/ceph/ceph-client.admin.40095.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40953.log 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.39075.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40095.log.gz 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.35173.log.gz 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39075.log.gz 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:20:27.494 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-client.admin.40953.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40953.log.gz 2024-03-25T18:20:27.497 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:20:27.497 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:20:27.497 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:20:27.497 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/valgrind/osd.7.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:20:27.497 INFO:teuthology.orchestra.run.smithi117.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:20:27.497 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/valgrind/osd.5.log: gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:20:27.497 INFO:teuthology.orchestra.run.smithi117.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:20:27.497 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:20:27.497 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/valgrind/osd.6.log: /var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35343.log 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.35343.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35343.log.gz 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr: 88.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35143.log 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-osd.7.log: /var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35434.log 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.35143.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.35434.log: gzip -5 86.6% -- replaced with /var/log/ceph/ceph-client.admin.35143.log.gz 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr: --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:20:27.498 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mgr.x.log: gzip 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35434.log.gz 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.35183.log 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35223.log 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.35183.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35303.log 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr: 87.3%/var/log/ceph/ceph-client.admin.35223.log: -- replaced with /var/log/ceph/ceph.log.gz 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35183.log.gz 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35223.log.gz 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.35303.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35385.log 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-mon.b.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.35303.log.gz 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35263.log 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/ceph-client.admin.35385.log: /var/log/ceph/ceph-client.admin.35263.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35385.log.gz 2024-03-25T18:20:27.499 INFO:teuthology.orchestra.run.smithi117.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.35263.log.gz 2024-03-25T18:20:27.528 INFO:teuthology.orchestra.run.smithi117.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:20:28.295 INFO:teuthology.orchestra.run.smithi037.stderr:/var/log/ceph/ceph-mon.c.log: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:20:28.351 INFO:teuthology.orchestra.run.smithi117.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:20:28.357 INFO:teuthology.orchestra.run.smithi117.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:20:28.416 INFO:teuthology.orchestra.run.smithi117.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:20:28.598 INFO:teuthology.orchestra.run.smithi037.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:20:28.647 INFO:teuthology.orchestra.run.smithi117.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:20:28.682 INFO:teuthology.orchestra.run.smithi037.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:20:28.690 INFO:teuthology.orchestra.run.smithi037.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:20:32.272 INFO:teuthology.orchestra.run.smithi037.stderr: 90.7% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:20:32.756 INFO:teuthology.orchestra.run.smithi117.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:20:32.758 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-03-25T18:20:32.758 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m5.298s 2024-03-25T18:20:32.758 INFO:teuthology.orchestra.run.smithi117.stderr:user 0m8.914s 2024-03-25T18:20:32.758 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m0.323s 2024-03-25T18:20:32.867 INFO:teuthology.orchestra.run.smithi037.stderr: 90.5% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:20:32.869 INFO:teuthology.orchestra.run.smithi037.stderr: 2024-03-25T18:20:32.869 INFO:teuthology.orchestra.run.smithi037.stderr:real 0m5.443s 2024-03-25T18:20:32.870 INFO:teuthology.orchestra.run.smithi037.stderr:user 0m14.224s 2024-03-25T18:20:32.870 INFO:teuthology.orchestra.run.smithi037.stderr:sys 0m0.538s 2024-03-25T18:20:32.870 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:20:32.870 DEBUG:teuthology.misc:Transferring archived files from smithi037:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265/remote/smithi037/log 2024-03-25T18:20:32.871 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:20:35.179 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265/remote/smithi117/log 2024-03-25T18:20:35.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:20:36.622 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:20:36.821 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c20eff9d809b485495a9045fbb1381f8 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:20:36.824 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:20:36.837 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:20:36.887 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:20:36.887 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:20:36.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:20:36.979 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:20:36.990 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:20:36.990 DEBUG:teuthology.orchestra.run.smithi037:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:20:36.993 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:20:37.007 INFO:teuthology.orchestra.run.smithi037.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:20:37.008 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:20:37.011 INFO:teuthology.orchestra.run.smithi037.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:20:37.012 INFO:teuthology.orchestra.run.smithi037.stdout:=============================================================================== 2024-03-25T18:20:37.012 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 13 +574us[ +574us] +/- 95ms 2024-03-25T18:20:37.012 INFO:teuthology.orchestra.run.smithi037.stdout:^* hv02.front.sepia.ceph.com 3 6 377 22 -1285us[-1278us] +/- 77ms 2024-03-25T18:20:37.012 INFO:teuthology.orchestra.run.smithi037.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 13 +726us[ +726us] +/- 66ms 2024-03-25T18:20:37.012 INFO:teuthology.orchestra.run.smithi037.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:20:37.013 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:20:37.013 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-03-25T18:20:37.013 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 12 +584us[ +584us] +/- 95ms 2024-03-25T18:20:37.014 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv02.front.sepia.ceph.com 3 6 377 18 -1267us[-1277us] +/- 77ms 2024-03-25T18:20:37.014 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 18 +752us[ +752us] +/- 66ms 2024-03-25T18:20:37.014 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:20:37.014 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:20:37.026 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:20:37.026 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:20:37.063 DEBUG:teuthology.orchestra.run.smithi037:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:20:37.133 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:20:37.199 DEBUG:teuthology.orchestra.run.smithi037:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:20:37.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:20:37.230 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:20:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:20:37.260 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:20:37.271 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:20:37.313 INFO:teuthology.task.internal:Duration was 7268.961729 seconds 2024-03-25T18:20:37.314 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:20:37.355 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:20:37.355 DEBUG:teuthology.orchestra.run.smithi037:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:20:37.358 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:20:37.400 INFO:teuthology.orchestra.run.smithi037.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:20:37.401 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:20:37.712 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:20:37.712 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi037.front.sepia.ceph.com 2024-03-25T18:20:37.712 DEBUG:teuthology.orchestra.run.smithi037:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:20:37.732 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-03-25T18:20:37.732 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:20:37.793 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:20:37.812 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:20:37.815 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:20:37.844 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:20:37.844 DEBUG:teuthology.orchestra.run.smithi037:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:20:37.885 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:20:37.991 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:20:38.001 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:20:38.001 DEBUG:teuthology.orchestra.run.smithi037:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:20:38.027 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:20:38.054 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:20:38.066 DEBUG:teuthology.orchestra.run.smithi037:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:20:38.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:20:38.093 INFO:teuthology.orchestra.run.smithi037.stdout:kernel.core_pattern = core 2024-03-25T18:20:38.120 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-03-25T18:20:38.142 DEBUG:teuthology.orchestra.run.smithi037:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:20:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:20:38.172 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:20:38.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:20:38.197 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:20:38.209 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:20:38.210 DEBUG:teuthology.misc:Transferring archived files from smithi037:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265/remote/smithi037 2024-03-25T18:20:38.210 DEBUG:teuthology.orchestra.run.smithi037:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:20:38.248 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265/remote/smithi117 2024-03-25T18:20:38.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:20:38.282 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:20:38.282 DEBUG:teuthology.orchestra.run.smithi037:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:20:38.285 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:20:38.332 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:20:38.344 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:20:38.344 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:20:38.380 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:20:38.380 DEBUG:teuthology.orchestra.run.smithi037:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:20:38.382 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:20:38.394 INFO:teuthology.orchestra.run.smithi037.stdout: 265365 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:20 /home/ubuntu/cephtest 2024-03-25T18:20:38.396 INFO:teuthology.orchestra.run.smithi117.stdout: 265480 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:20 /home/ubuntu/cephtest 2024-03-25T18:20:38.397 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:20:38.419 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:20:38.522 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:20:38.548 DEBUG:teuthology.nuke:shortname: smithi037 2024-03-25T18:20:38.548 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:20:38.569 DEBUG:teuthology.nuke:shortname: smithi117 2024-03-25T18:20:38.569 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:20:38.575 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi037.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:11:20.937895', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBMNdzPNwpNxL8sLSxCBb3qYaQUyuKLhg/+dU6nOMhGbC99bahPbGjo7T9lDbLNGoTdxPzhvw8f2JmpIjH68p7bY='} 2024-03-25T18:20:38.587 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621265', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:11:20.939544', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAx0A2u20CfmOD6mD1RM83qmHc3FrWuU9cHfiZY/crmiPZ9xhA8kxQyqM2MmBawk1+PAyifrIOL2XSXvt52zQjE='} 2024-03-25T18:20:38.620 INFO:teuthology.orchestra.console.smithi037:Power off 2024-03-25T18:20:38.620 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:20:38.637 INFO:teuthology.orchestra.console.smithi117:Power off 2024-03-25T18:20:38.638 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:20:38.650 DEBUG:teuthology.orchestra.console.smithi037:power off output: Chassis Power Control: Down/Off 2024-03-25T18:20:38.651 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:20:38.661 DEBUG:teuthology.orchestra.console.smithi117:power off output: Chassis Power Control: Down/Off 2024-03-25T18:20:38.662 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:20:38.673 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2024-03-25T18:20:38.684 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-25T18:20:42.674 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:20:42.685 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:20:42.785 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2024-03-25T18:20:42.794 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-03-25T18:20:46.789 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:20:46.796 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:20:46.898 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is on 2024-03-25T18:20:46.906 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is off 2024-03-25T18:20:47.007 INFO:teuthology.orchestra.console.smithi117:Power off completed 2024-03-25T18:20:50.904 DEBUG:teuthology.orchestra.console.smithi037:pexpect command: ipmitool -H smithi037.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:20:51.014 DEBUG:teuthology.orchestra.console.smithi037:check power output: Chassis Power is off 2024-03-25T18:20:51.115 INFO:teuthology.orchestra.console.smithi037:Power off completed 2024-03-25T18:20:51.219 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7268.961728811264 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c20eff9d809b485495a9045fbb1381f8 status: fail success: false 2024-03-25T18:20:51.220 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:20:51.561 INFO:teuthology.run:FAIL